预先训练的语言模型（LMS）通常逻辑地扭转或以组成方式概括。最近的工作表明，结合外部实体知识可以提高LMS的能力和推广。然而，明确提供实体抽象的效果仍然不清楚，特别是在最近的研究表明，预先训练的LMS已经在其参数中编码了一些知识。我们研究将实体型抽象的实用程序融入预先训练的变压器，并在需要不同形式的逻辑推理的四个NLP任务上测试这些方法：（1）与基于文本的关系推理（CLUTRR）的组成语言理解，（2）绑架推理（校对者），（3）多跳问题应答（HotpotQA），和（4）会话问题应答（COQA）。我们提出并经验探索了三种方法来添加此类抽象：（i）作为附加输入嵌入式，（ii）作为编码的单独序列，（iii）作为模型的辅助预测任务。总体而言，我们的分析表明，具有抽象实体知识的模型比没有它更好。然而，我们的实验还表明，强烈的益处取决于所使用的技术和手头的任务。与基线模型相比，最佳抽象意识模型分别达到了88.8％和91.8％的总精度，分别在CLUTRR和校对者上实现了62.3％和89.8％。此外，抽象感知模型在插值和外推设置中显示出改善的组成概括。然而，对于热杆菌和CoQA，我们发现F1分数平均仅提高0.5％。我们的结果表明，明确抽象的好处在正式定义的逻辑推理设置中需要许多推理跳跃，但指向它对具有较少正式逻辑结构的NLP任务不利的概念。

从头开始解决复杂问题通常是有挑战性的，但如果我们可以访问其解决方案的其他类似问题，则更容易 - 一种称为基于案例的推理（CBR）的范式。我们提出了一种神经象征性的CBR方法（CBR-KBQA），用于在大知识库上应答。 CBR-KBQA由非参数内存组成，该内存存储案例（问题和逻辑表单）和参数模型，该参数模型可以通过检索与其相关的案例来为新问题生成逻辑表单。在包含复杂问题的几个KBQA数据集上，CBR-KBQA实现了竞争性能。例如，在ComplexWebQuestions数据集上，CBR-KBQA以11 \％的准确度优于当前最新状态。此外，我们表明CBR-KBQA能够使用新案例\ EMPH {没有}任何进一步的培训：通过在案例存储器中纳入一些人类标记的示例，CBR-KBQA能够成功地生成包含未经看线KB实体的逻辑表格以及关系。

Transformers have been shown to be able to perform deductive reasoning on a logical rulebase containing rules and statements written in English natural language. While the progress is promising, it is currently unclear if these models indeed perform logical reasoning by understanding the underlying logical semantics in the language. To this end, we propose RobustLR, a suite of evaluation datasets that evaluate the robustness of these models to minimal logical edits in rulebases and some standard logical equivalence conditions. In our experiments with RoBERTa and T5, we find that the models trained in prior works do not perform consistently on the different perturbations in RobustLR, thus showing that the models are not robust to the proposed logical perturbations. Further, we find that the models find it especially hard to learn logical negation and disjunction operators. Overall, using our evaluation sets, we demonstrate some shortcomings of the deductive reasoning-based language models, which can eventually help towards designing better models for logical reasoning over natural language. All the datasets and code base have been made publicly available.

使用诸如BERT，ELMO和FLAIR等模型建模上下文信息的成立具有显着改善了文字的表示学习。它还给出了几乎每个NLP任务机器翻译，文本摘要和命名实体识别的Sota结果，以命名为少。在这项工作中，除了使用这些主导的上下文感知的表示之外，我们还提出了一种用于命名实体识别（NER）的知识意识表示学习（KARL）网络。我们讨论了利用现有方法在纳入世界知识方面的挑战，并展示了如何利用我们所提出的方法来克服这些挑战。 KARL基于变压器编码器，该变压器编码器利用表示为事实三元组的大知识库，将它们转换为图形上下文，并提取驻留在内部的基本实体信息以生成用于特征增强的上下文化三联表示。实验结果表明，使用卡尔的增强可以大大提升我们的内部系统的性能，并在三个公共网络数据集中的文献中的现有方法，即Conll 2003，Conll ++和Ontonotes V5实现了比文献中现有方法的显着更好的结果。我们还观察到更好的概括和应用于从Karl上看不见的实体的真实环境。

大型基于变压器的预训练的语言模型在各种知识密集的任务上取得了令人印象深刻的表现，并可以在其参数中捕获事实知识。我们认为，考虑到不断增长的知识和资源需求，在模型参数中存储大量知识是亚最佳选择。我们认为，更有效的替代方法是向模型提供对上下文相关的结构化知识的明确访问，并训练它以使用该知识。我们提出了LM核 - 实现这一目标的一般框架 - 允许从外部知识源对语言模型培训的\ textit {解耦}，并允许后者更新而不会影响已经训练的模型。实验结果表明，LM核心获得外部知识，在知识探索任务上的最先进的知识增强语言模型中实现了重要而强大的优于性能。可以有效处理知识更新；并在两个下游任务上表现良好。我们还提出了一个彻底的错误分析，突出了LM核的成功和失败。

与伯特（Bert）等语言模型相比，已证明知识增强语言表示的预培训模型在知识基础构建任务（即〜关系提取）中更有效。这些知识增强的语言模型将知识纳入预训练中，以生成实体或关系的表示。但是，现有方法通常用单独的嵌入表示每个实体。结果，这些方法难以代表播出的实体和大量参数，在其基础代币模型之上（即〜变压器），必须使用，并且可以处理的实体数量为由于内存限制，实践限制。此外，现有模型仍然难以同时代表实体和关系。为了解决这些问题，我们提出了一个新的预培训模型，该模型分别从图书中学习实体和关系的表示形式，并分别在文本中跨越跨度。通过使用SPAN模块有效地编码跨度，我们的模型可以代表实体及其关系，但所需的参数比现有模型更少。我们通过从Wikipedia中提取的知识图对我们的模型进行了预训练，并在广泛的监督和无监督的信息提取任务上进行了测试。结果表明，我们的模型比基线学习对实体和关系的表现更好，而在监督的设置中，微调我们的模型始终优于罗伯塔，并在信息提取任务上取得了竞争成果。

在这项工作中，我们专注于从自然语言问题中生成SPARQL查询的任务，然后可以在知识图（kgs）上执行。我们假设已经提供了黄金实体和关系，其余的任务是与Sparql词汇一起以正确的顺序排列它们，并输入令牌以产生正确的SPARQL查询。到目前为止，尚未对此任务进行深入探索，因此我们使用BERT嵌入的BART，T5和PGN（指针发电机网络）进行了深入探讨，因此，请在PLM ERA中寻找此任务的新基础，在dbpedia和wikidata kgs上。我们表明T5需要特殊的输入令牌化，但是在LC-Quad 1.0和LC-Quad 2.0数据集上产生最先进的性能，并且从以前的工作中优于特定于任务的模型。此外，这些方法可以为问题进行语义解析，以使输入的一部分需要复制到输出查询，从而在KG语义解析中启用新的范式。

在这项工作中，我们探索如何学习专用的语言模型，旨在学习从文本文件中学习关键词的丰富表示。我们在判别和生成设置中进行预训练变压器语言模型（LMS）的不同掩蔽策略。在歧视性设定中，我们引入了一种新的预训练目标 - 关键边界，用替换（kbir）infifiling，在使用Kbir预先训练的LM进行微调时显示出在Sota上的性能（F1中高达9.26点）的大量增益关键酶提取的任务。在生成设置中，我们为BART - 键盘介绍了一个新的预训练设置，可再现与CATSeq格式中的输入文本相关的关键字，而不是Denoised原始输入。这也导致在关键词中的性能（F1 @ M）中的性能（高达4.33点），用于关键正版生成。此外，我们还微调了在命名实体识别（ner），问题应答（qa），关系提取（重新），抽象摘要和达到与SOTA的可比性表现的预训练的语言模型，表明学习丰富的代表关键词确实有利于许多其他基本的NLP任务。

Transfer learning, where a model is first pre-trained on a data-rich task before being finetuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new "Colossal Clean Crawled Corpus", we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.

Entities, as important carriers of real-world knowledge, play a key role in many NLP tasks. We focus on incorporating entity knowledge into an encoder-decoder framework for informative text generation. Existing approaches tried to index, retrieve, and read external documents as evidence, but they suffered from a large computational overhead. In this work, we propose an encoder-decoder framework with an entity memory, namely EDMem. The entity knowledge is stored in the memory as latent representations, and the memory is pre-trained on Wikipedia along with encoder-decoder parameters. To precisely generate entity names, we design three decoding methods to constrain entity generation by linking entities in the memory. EDMem is a unified framework that can be used on various entity-intensive question answering and generation tasks. Extensive experimental results show that EDMem outperforms both memory-based auto-encoder models and non-memory encoder-decoder models.

大型语言模型在各种任务上显示出令人印象深刻的几次结果。但是，当知识是此类结果的关键时，就像问题回答和事实检查之类的任务一样，似乎需要存储知识的大量参数计数。众所周知，检索增强模型可以在不需要多个参数的情况下在知识密集的任务上表现出色，但是目前尚不清楚它们是否在几个弹药设置中工作。在这项工作中，我们介绍了地图集，这是一个经过精心设计和预先训练的增强语言模型，能够通过很少的培训示例学习知识密集型任务。我们对包括MMLU，苏格兰短裙和归类等各种任务进行评估，并研究文档索引内容的影响，表明它可以很容易地进行更新。值得注意的是，在自然问题上仅使用64个示例在自然问题上达到超过42 \％的准确性，尽管参数少了50倍，但比540B参数模型的表现优于540b参数模型。

Natural Language Processing (NLP) has been revolutionized by the use of Pre-trained Language Models (PLMs) such as BERT. Despite setting new records in nearly every NLP task, PLMs still face a number of challenges including poor interpretability, weak reasoning capability, and the need for a lot of expensive annotated data when applied to downstream tasks. By integrating external knowledge into PLMs, \textit{\underline{K}nowledge-\underline{E}nhanced \underline{P}re-trained \underline{L}anguage \underline{M}odels} (KEPLMs) have the potential to overcome the above-mentioned limitations. In this paper, we examine KEPLMs systematically through a series of studies. Specifically, we outline the common types and different formats of knowledge to be integrated into KEPLMs, detail the existing methods for building and evaluating KEPLMS, present the applications of KEPLMs in downstream tasks, and discuss the future research directions. Researchers will benefit from this survey by gaining a quick and comprehensive overview of the latest developments in this field.

We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it. Span-BERT consistently outperforms BERT and our better-tuned baselines, with substantial gains on span selection tasks such as question answering and coreference resolution. In particular, with the same training data and model size as BERT large , our single model obtains 94.6% and 88.7% F1 on SQuAD 1.1 and 2.0 respectively. We also achieve a new state of the art on the OntoNotes coreference resolution task (79.6% F1), strong performance on the TACRED relation extraction benchmark, and even gains on GLUE. 1 * Equal contribution. 1 Our code and pre-trained models are available at https://github.com/facebookresearch/ SpanBERT.

Recent progress in pretraining language models on large textual corpora led to a surge of improvements for downstream NLP tasks. Whilst learning linguistic knowledge, these models may also be storing relational knowledge present in the training data, and may be able to answer queries structured as "fillin-the-blank" cloze statements. Language models have many advantages over structured knowledge bases: they require no schema engineering, allow practitioners to query about an open class of relations, are easy to extend to more data, and require no human supervision to train. We present an in-depth analysis of the relational knowledge already present (without fine-tuning) in a wide range of state-of-theart pretrained language models. We find that (i) without fine-tuning, BERT contains relational knowledge competitive with traditional NLP methods that have some access to oracle knowledge, (ii) BERT also does remarkably well on open-domain question answering against a supervised baseline, and (iii) certain types of factual knowledge are learned much more readily than others by standard language model pretraining approaches. The surprisingly strong ability of these models to recall factual knowledge without any fine-tuning demonstrates their potential as unsupervised open-domain QA systems. The code to reproduce our analysis is available at https: //github.com/facebookresearch/LAMA.

Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue and approaches to compression. We then outline directions for future research.

问题答案（QA）是自然语言处理中最具挑战性的最具挑战性的问题之一（NLP）。问答（QA）系统试图为给定问题产生答案。这些答案可以从非结构化或结构化文本生成。因此，QA被认为是可以用于评估文本了解系统的重要研究区域。大量的QA研究致力于英语语言，调查最先进的技术和实现最先进的结果。然而，由于阿拉伯QA中的研究努力和缺乏大型基准数据集，在阿拉伯语问答进展中的研究努力得到了很大速度的速度。最近许多预先接受的语言模型在许多阿拉伯语NLP问题中提供了高性能。在这项工作中，我们使用四个阅读理解数据集来评估阿拉伯QA的最先进的接种变压器模型，它是阿拉伯语 - 队，ArcD，AQAD和TYDIQA-GoldP数据集。我们微调并比较了Arabertv2基础模型，ArabertV0.2大型型号和ARAElectra模型的性能。在最后，我们提供了一个分析，了解和解释某些型号获得的低绩效结果。

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a;Radford et al., 2018), BERT is designed to pretrain deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be finetuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial taskspecific architecture modifications.BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

Neural language representation models such as BERT pre-trained on large-scale corpora can well capture rich semantic patterns from plain text, and be fine-tuned to consistently improve the performance of various NLP tasks. However, the existing pre-trained language models rarely consider incorporating knowledge graphs (KGs), which can provide rich structured knowledge facts for better language understanding. We argue that informative entities in KGs can enhance language representation with external knowledge. In this paper, we utilize both large-scale textual corpora and KGs to train an enhanced language representation model (ERNIE), which can take full advantage of lexical, syntactic, and knowledge information simultaneously. The experimental results have demonstrated that ERNIE achieves significant improvements on various knowledge-driven tasks, and meanwhile is comparable with the state-of-the-art model BERT on other common NLP tasks. The source code and experiment details of this paper can be obtained from https:// github.com/thunlp/ERNIE.

最近的研究表明，自然语言理解中的系统概括仍然是最先进的神经模型（如变形金刚和图形神经网络）的挑战。为了解决这一挑战，我们提出了边缘变压器，这是一种新的模型，将灵感与基于规则的符号AI结合起来。边缘变压器中的第一个关键思想是将矢量状态与每个边缘相关联，即使用每对输入节点 - 与每个节点相对，因为它在变压器模型中完成。第二重要创新是一个三角形关注机制，以通过从逻辑编程的统一启发的方式更新边缘表示。我们在关系推理，语义解析和依赖性解析中评估边缘变压器上的成分泛化基准。在所有三种设置中，边缘变压器优于关系感知，通用和古典变压器基线。

Recent progress in pre-trained neural language models has significantly improved the performance of many natural language processing (NLP) tasks. In this paper we propose a new model architecture DeBERTa (Decoding-enhanced BERT with disentangled attention) that improves the BERT and RoBERTa models using two novel techniques. The first is the disentangled attention mechanism, where each word is represented using two vectors that encode its content and position, respectively, and the attention weights among words are computed using disentangled matrices on their contents and relative positions, respectively. Second, an enhanced mask decoder is used to incorporate absolute positions in the decoding layer to predict the masked tokens in model pre-training. In addition, a new virtual adversarial training method is used for fine-tuning to improve models' generalization. We show that these techniques significantly improve the efficiency of model pre-training and the performance of both natural language understand (NLU) and natural langauge generation (NLG) downstream tasks. Compared to RoBERTa-Large, a DeBERTa model trained on half of the training data performs consistently better on a wide range of NLP tasks, achieving improvements on MNLI by +0.9% (90.2% vs. 91.1%), on SQuAD v2.0 by +2.3% (88.4% vs. 90.7%) and RACE by +3.6% (83.2% vs. 86.8%). Notably, we scale up DeBERTa by training a larger version that consists of 48 Transform layers with 1.5 billion parameters. The significant performance boost makes the single DeBERTa model surpass the human performance on the SuperGLUE benchmark (Wang et al., 2019a) for the first time in terms of macro-average score (89.9 versus 89.8), and the ensemble DeBERTa model sits atop the SuperGLUE leaderboard as of January 6, 2021, outperforming the human baseline by a decent margin (90.3 versus 89.8). The pre-trained DeBERTa models and the source code were released at: https://github.com/microsoft/DeBERTa 1 .