最近的工作表明，（1）增加输入长度或（2）增加模型大小可以提高基于变压器的神经模型的性能。在本文中，我们提出了一个名为Longt5的新模型，我们探讨了同时缩放输入长度和模型大小的效果。具体而言，我们综合了从长输入变压器（ETC）的关注思路，并采用了从摘要预训练（PEGASU）的预训练策略进入可扩展的T5架构。结果是我们称之为{\ EM瞬态全球}（TGLOBAL）的新关注机制，这些机制是模仿等本地/全球注意力机制，但不需要额外的侧面输入。我们能够实现最先进的结果，以若干摘要任务，优于问题应答任务的原始T5模型。

我们提出了一项实证研究，以适应现有的经过验证的文本对文本模型，以备长期输入。通过沿预训练管道的三个轴的全面研究 - 模型架构，优化目标和训练式语料库，我们提出了一种有效的食谱，以从现有的短篇小说模型中构建长篇小说模型。具体而言，我们用汇总仪的块关注替换了变压器中的全部注意力，并使用蒙版的跨度预测任务为模型预算，长度不同。就训练训练的语料库而言，我们发现，与使用通常在其域覆盖范围中通常受到限制的现有长文档语料库相比，使用大型开放域语料库的随机串联的短篇小说可以提高性能。通过这些发现，我们建立了一个长篇文本模型，该模型可以在长篇文本质量检查任务上实现竞争性能，并在五个长文本摘要数据集上建立新的最新技术，通常优于先前的方法，具有较大的模型大小。

尽管经过验证的大型变压器模型已被证明具有很高的能力解决自然语言任务，但处理长序列输入仍然是一个重大挑战。这样的任务之一就是长输入摘要，其中输入比大多数预验证的模型的最大输入上下文更长。通过一系列广泛的实验，我们研究了哪些模型架构变化和预处理范式可以最有效地适应经过预定的变压器以进行长输入摘要。我们发现，带有全局编码器代币的交错，块状变压器可以达到良好的性能和效率平衡，并且在长序列上有意义地改善了下游摘要性能。根据我们的发现，我们介绍了Pegasus-X，这是Pegasus模型的扩展，并具有额外的长输入预处理，以处理最多16K令牌的输入。 Pegasus-X在长输入摘要任务上实现了强劲的性能，与更大的模型相当，同时添加了很少的其他参数，并且不需要模型并行训练。

Transfer learning, where a model is first pre-trained on a data-rich task before being finetuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new "Colossal Clean Crawled Corpus", we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.

We present BART, a denoising autoencoder for pretraining sequence-to-sequence models. BART is trained by ( 1) corrupting text with an arbitrary noising function, and (2) learning a model to reconstruct the original text. It uses a standard Tranformer-based neural machine translation architecture which, despite its simplicity, can be seen as generalizing BERT (due to the bidirectional encoder), GPT (with the left-to-right decoder), and many other more recent pretraining schemes. We evaluate a number of noising approaches, finding the best performance by both randomly shuffling the order of the original sentences and using a novel in-filling scheme, where spans of text are replaced with a single mask token. BART is particularly effective when fine tuned for text generation but also works well for comprehension tasks. It matches the performance of RoBERTa with comparable training resources on GLUE and SQuAD, achieves new stateof-the-art results on a range of abstractive dialogue, question answering, and summarization tasks, with gains of up to 6 ROUGE. BART also provides a 1.1 BLEU increase over a back-translation system for machine translation, with only target language pretraining. We also report ablation experiments that replicate other pretraining schemes within the BART framework, to better measure which factors most influence end-task performance.

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a;Radford et al., 2018), BERT is designed to pretrain deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be finetuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial taskspecific architecture modifications.BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

在这项工作中，我们探索如何学习专用的语言模型，旨在学习从文本文件中学习关键词的丰富表示。我们在判别和生成设置中进行预训练变压器语言模型（LMS）的不同掩蔽策略。在歧视性设定中，我们引入了一种新的预训练目标 - 关键边界，用替换（kbir）infifiling，在使用Kbir预先训练的LM进行微调时显示出在Sota上的性能（F1中高达9.26点）的大量增益关键酶提取的任务。在生成设置中，我们为BART - 键盘介绍了一个新的预训练设置，可再现与CATSeq格式中的输入文本相关的关键字，而不是Denoised原始输入。这也导致在关键词中的性能（F1 @ M）中的性能（高达4.33点），用于关键正版生成。此外，我们还微调了在命名实体识别（ner），问题应答（qa），关系提取（重新），抽象摘要和达到与SOTA的可比性表现的预训练的语言模型，表明学习丰富的代表关键词确实有利于许多其他基本的NLP任务。

查询聚焦的文本摘要（QFTS）任务旨在构建基于给定查询的文本文档摘要的构建系统。解决此任务的关键挑战是缺乏培训摘要模型的大量标记数据。在本文中，我们通过探索一系列域适应技术来解决这一挑战。鉴于最近在广泛的自然语言处理任务中进行预先接受的变压器模型的成功，我们利用此类模型为单文档和多文件方案的QFTS任务产生抽象摘要。对于域适应，我们使用预先训练的变压器的摘要模型应用了各种技术，包括转移学习，弱监督学习和远程监督。六个数据集的广泛实验表明，我们所提出的方法非常有效地为QFTS任务产生抽象摘要，同时在一组自动和人类评估指标上设置新的最先进的结果。

Entities, as important carriers of real-world knowledge, play a key role in many NLP tasks. We focus on incorporating entity knowledge into an encoder-decoder framework for informative text generation. Existing approaches tried to index, retrieve, and read external documents as evidence, but they suffered from a large computational overhead. In this work, we propose an encoder-decoder framework with an entity memory, namely EDMem. The entity knowledge is stored in the memory as latent representations, and the memory is pre-trained on Wikipedia along with encoder-decoder parameters. To precisely generate entity names, we design three decoding methods to constrain entity generation by linking entities in the memory. EDMem is a unified framework that can be used on various entity-intensive question answering and generation tasks. Extensive experimental results show that EDMem outperforms both memory-based auto-encoder models and non-memory encoder-decoder models.

诸如学术文章和商业报告之类的长期文件一直是详细说明重要问题和需要额外关注的复杂主题的标准格式。自动汇总系统可以有效地将长文档置于简短而简洁的文本中，以封装最重要的信息，从而在帮助读者的理解中很重要。最近，随着神经体系结构的出现，已经做出了重大的研究工作，以推动自动文本摘要系统，以及有关将这些系统扩展到长期文档领域的挑战的大量研究。在这项调查中，我们提供了有关长期文档摘要的研究的全面概述，以及其研究环境的三个主要组成部分的系统评估：基准数据集，汇总模型和评估指标。对于每个组成部分，我们在长期汇总的背景下组织文献，并进行经验分析，以扩大有关当前研究进度的观点。实证分析包括一项研究基准数据集的内在特征，摘要模型的多维分析以及摘要评估指标的综述。根据总体发现，我们通过提出可能在这个快速增长的领域中提出未来探索的方向来得出结论。

基于变压器的预审前的语言模型（LMS）在自然语言的理解中无处不在，但由于其二次复杂性，无法应用于故事，科学文章和长文档等长序列。尽管已经提出了无数有效的变压器变体，但它们通常是基于需要从头开始的昂贵预处理的自定义实现。在这项工作中，我们提出了雪橇：滑动编码器和解码器，这是一种处理长序列的简单方法，可以重新使用和利用经过战斗测试的短文本预处理的LMS。具体而言，我们将输入分配到重叠的块中，用短文本LM编码器编码每个块，然后使用预审计的解码器将信息融合到跨块（Fusion-In-In-In-In-indecoder）之间。我们通过受控实验说明，雪橇提供了一种可行的策略，可以长期理解并评估我们在卷轴上的方法，这是一个基准，该基准在各种语言理解任务中具有七个数据集。我们发现，雪橇与高达50倍的专业型号具有竞争力，并且需要专用且昂贵的预处理步骤。

对话是人类沟通与合作的重要组成部分。现有研究主要关注一对一时尚的短对话情景。然而，现实世界中的多人互动，例如会议或访谈，经常超过几千个字。仍然缺乏相应的研究和强大的工具来了解和处理这么长的对话。因此，在这项工作中，我们为长时间对话理解和总结提供了预先培训框架。考虑到长期交谈的性质，我们提出了一种基于窗口的去噪方法，用于生成预训练。对于对话框，它损坏了一个带有对话激发灵感噪声的文本窗口，并指导模型基于剩余对话的内容来重建此窗口。此外，为了更长的输入，我们增加了稀疏关注模型，这些模型以混合方式与传统的关注相结合。我们在长对话的五个数据集进行广泛的实验，涵盖对话摘要的任务，抽象问题回答和主题分割。实验，我们表明，我们的预先训练的模型DialogLM显着超越了数据集和任务的最先进的模型。我们的GitHub存储库（HTTPS:/github.com/microsoft/dialoglm上有源代码和所有预先训练的型号。

We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it. Span-BERT consistently outperforms BERT and our better-tuned baselines, with substantial gains on span selection tasks such as question answering and coreference resolution. In particular, with the same training data and model size as BERT large , our single model obtains 94.6% and 88.7% F1 on SQuAD 1.1 and 2.0 respectively. We also achieve a new state of the art on the OntoNotes coreference resolution task (79.6% F1), strong performance on the TACRED relation extraction benchmark, and even gains on GLUE. 1 * Equal contribution. 1 Our code and pre-trained models are available at https://github.com/facebookresearch/ SpanBERT.

The task of automatic text summarization produces a concise and fluent text summary while preserving key information and overall meaning. Recent approaches to document-level summarization have seen significant improvements in recent years by using models based on the Transformer architecture. However, the quadratic memory and time complexities with respect to the sequence length make them very expensive to use, especially with long sequences, as required by document-level summarization. Our work addresses the problem of document-level summarization by studying how efficient Transformer techniques can be used to improve the automatic summarization of very long texts. In particular, we will use the arXiv dataset, consisting of several scientific papers and the corresponding abstracts, as baselines for this work. Then, we propose a novel retrieval-enhanced approach based on the architecture which reduces the cost of generating a summary of the entire document by processing smaller chunks. The results were below the baselines but suggest a more efficient memory a consumption and truthfulness.

Recent progress in pre-trained neural language models has significantly improved the performance of many natural language processing (NLP) tasks. In this paper we propose a new model architecture DeBERTa (Decoding-enhanced BERT with disentangled attention) that improves the BERT and RoBERTa models using two novel techniques. The first is the disentangled attention mechanism, where each word is represented using two vectors that encode its content and position, respectively, and the attention weights among words are computed using disentangled matrices on their contents and relative positions, respectively. Second, an enhanced mask decoder is used to incorporate absolute positions in the decoding layer to predict the masked tokens in model pre-training. In addition, a new virtual adversarial training method is used for fine-tuning to improve models' generalization. We show that these techniques significantly improve the efficiency of model pre-training and the performance of both natural language understand (NLU) and natural langauge generation (NLG) downstream tasks. Compared to RoBERTa-Large, a DeBERTa model trained on half of the training data performs consistently better on a wide range of NLP tasks, achieving improvements on MNLI by +0.9% (90.2% vs. 91.1%), on SQuAD v2.0 by +2.3% (88.4% vs. 90.7%) and RACE by +3.6% (83.2% vs. 86.8%). Notably, we scale up DeBERTa by training a larger version that consists of 48 Transform layers with 1.5 billion parameters. The significant performance boost makes the single DeBERTa model surpass the human performance on the SuperGLUE benchmark (Wang et al., 2019a) for the first time in terms of macro-average score (89.9 versus 89.8), and the ensemble DeBERTa model sits atop the SuperGLUE leaderboard as of January 6, 2021, outperforming the human baseline by a decent margin (90.3 versus 89.8). The pre-trained DeBERTa models and the source code were released at: https://github.com/microsoft/DeBERTa 1 .

本文介绍了Z-Code ++，这是一种针对抽象文本摘要优化的新的预训练的语言模型。该模型使用三种技术扩展了艺术编码器模型的状态。首先，我们使用两阶段的预训练过程来改善模型在低资源摘要任务上的性能。该模型首先是使用文本语料库进行语言理解的预先培训的，然后在汇总语料库中不断预先培训，以进行基础文本生成。其次，我们用分离的注意力层代替编码器中的自我发项层，其中每个单词都使用两个向量分别代表其内容和位置。第三，我们使用融合编码器，这是一种以层次方式编码长序列的简单而有效的方法。 Z-Code ++在13个文本摘要任务中的9个跨5种语言中创建了新的艺术状态。我们的模型的参数有效，因为它的表现优于XSUM上600倍较大的Palm-540b，并且在Samsum上的易经的200倍GPT3-175B较大。在零射击和少量设置中，我们的模型大大优于竞争模型。

大型语言模型在各种任务上显示出令人印象深刻的几次结果。但是，当知识是此类结果的关键时，就像问题回答和事实检查之类的任务一样，似乎需要存储知识的大量参数计数。众所周知，检索增强模型可以在不需要多个参数的情况下在知识密集的任务上表现出色，但是目前尚不清楚它们是否在几个弹药设置中工作。在这项工作中，我们介绍了地图集，这是一个经过精心设计和预先训练的增强语言模型，能够通过很少的培训示例学习知识密集型任务。我们对包括MMLU，苏格兰短裙和归类等各种任务进行评估，并研究文档索引内容的影响，表明它可以很容易地进行更新。值得注意的是，在自然问题上仅使用64个示例在自然问题上达到超过42 \％的准确性，尽管参数少了50倍，但比540B参数模型的表现优于540b参数模型。

长文件摘要是自然语言处理领域的重要且艰巨的任务。良好的长文件摘要表现揭示了模型对人类语言的理解。目前，大多数研究侧重于如何修改变压器的注意机制，实现更高的胭脂分数。数据预处理和后处理的研究相对较少。在本文中，我们使用两个预处理方法和后处理方法，并分析了这些方法对各种长文件摘要模型的影响。

This paper presents a new UNIfied pre-trained Language Model (UNILM) that can be fine-tuned for both natural language understanding and generation tasks. The model is pre-trained using three types of language modeling tasks: unidirectional, bidirectional, and sequence-to-sequence prediction. The unified modeling is achieved by employing a shared Transformer network and utilizing specific self-attention masks to control what context the prediction conditions on. UNILM compares favorably with BERT on the GLUE benchmark, and the SQuAD 2.0 and CoQA question answering tasks. Moreover, UNILM achieves new state-ofthe-art results on five natural language generation datasets, including improving the CNN/DailyMail abstractive summarization ROUGE-L to 40.51 (2.04 absolute improvement), the Gigaword abstractive summarization ROUGE-L to 35.75 (0.86 absolute improvement), the CoQA generative question answering F1 score to 82.5 (37.1 absolute improvement), the SQuAD question generation BLEU-4 to 22.12 (3.75 absolute improvement), and the DSTC7 document-grounded dialog response generation NIST-4 to 2.67 (human performance is 2.65). The code and pre-trained models are available at https://github.com/microsoft/unilm. * Equal contribution. † Contact person.

传达相关和忠实信息的能力对于有条件生成的许多任务至关重要，但对于神经SEQ-seq seq模型仍然难以捉摸，这些模型的输出通常显示出幻觉，并且无法正确涵盖重要细节。在这项工作中，我们主张规划作为有用的中间表示，以使有条件的一代减少不透明和扎根。我们的作品提出了将文本计划作为一系列提问（QA）对的新概念化。我们用QA蓝图作为内容选择（即〜说什么）和计划（即〜按什么顺序）来增强现有数据集（例如，用于摘要）。我们通过利用最先进的问题生成技术并将输入输出对自动获取蓝图，并将其转换为输入 - 蓝图输出输出元组。我们开发了基于变压器的模型，每个模型都在它们如何将蓝图合并到生成的输出中（例如，作为全局计划或迭代）。跨指标和数据集的评估表明，蓝图模型比不采取计划并允许对生成输出进行更严格控制的替代方案更为事实。