Contextual word representations derived from large-scale neural language models are successful across a diverse set of NLP tasks, suggesting that they encode useful and transferable features of language. To shed light on the linguistic knowledge they capture, we study the representations produced by several recent pretrained contextualizers (variants of ELMo, the OpenAI transformer language model, and BERT) with a suite of seventeen diverse probing tasks. We find that linear models trained on top of frozen contextual representations are competitive with state-of-the-art task-specific models in many cases, but fail on tasks requiring fine-grained linguistic knowledge (e.g., conjunct identification). To investigate the transferability of contextual word representations, we quantify differences in the transferability of individual layers within contextualizers, especially between recurrent neural networks (RNNs) and transformers. For instance, higher layers of RNNs are more taskspecific, while transformer layers do not exhibit the same monotonic trend. In addition, to better understand what makes contextual word representations transferable, we compare language model pretraining with eleven supervised pretraining tasks. For any given task, pretraining on a closely related task yields better performance than language model pretraining (which is better on average) when the pretraining dataset is fixed. However, language model pretraining on more data gives the best results.

Contextualized representation models such as ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2018) have recently achieved state-of-the-art results on a diverse array of downstream NLP tasks. Building on recent token-level probing work, we introduce a novel edge probing task design and construct a broad suite of sub-sentence tasks derived from the traditional structured NLP pipeline. We probe word-level contextual representations from four recent models and investigate how they encode sentence structure across a range of syntactic, semantic, local, and long-range phenomena. We find that existing models trained on language modeling and translation produce strong representations for syntactic phenomena, but only offer comparably small improvements on semantic tasks over a non-contextual baseline.

Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue and approaches to compression. We then outline directions for future research.

基于变压器的语言模型最近在许多自然语言任务中取得了显着的结果。但是，通常通过利用大量培训数据来实现排行榜的性能，并且很少通过将明确的语言知识编码为神经模型。这使许多人质疑语言学对现代自然语言处理的相关性。在本文中，我介绍了几个案例研究，以说明理论语言学和神经语言模型仍然相互关联。首先，语言模型通过提供一个客观的工具来测量语义距离，这对语言学家很有用，语义距离很难使用传统方法。另一方面，语言理论通过提供框架和数据源来探究我们的语言模型，以了解语言理解的特定方面，从而有助于语言建模研究。本论文贡献了三项研究，探讨了语言模型中语法 - 听觉界面的不同方面。在论文的第一部分中，我将语言模型应用于单词类灵活性的问题。我将Mbert作为语义距离测量的来源，我提供了有利于将单词类灵活性分析为方向过程的证据。在论文的第二部分中，我提出了一种方法来测量语言模型中间层的惊奇方法。我的实验表明，包含形态句法异常的句子触发了语言模型早期的惊喜，而不是语义和常识异常。最后，在论文的第三部分中，我适应了一些心理语言学研究，以表明语言模型包含了论证结构结构的知识。总而言之，我的论文在自然语言处理，语言理论和心理语言学之间建立了新的联系，以为语言模型的解释提供新的观点。

在NLP社区中有一个正在进行的辩论，无论现代语言模型是否包含语言知识，通过所谓的探针恢复。在本文中，我们研究了语言知识是否是现代语言模型良好表现的必要条件，我们称之为\ Texit {重新发现假设}。首先，我们展示了语言模型，这是显着压缩的，但在预先磨普目标上表现良好，以便在语言结构探讨时保持良好的分数。这一结果支持重新发现的假设，并导致我们的论文的第二款贡献：一个信息 - 理论框架，与语言建模目标相关。该框架还提供了测量语言信息对字词预测任务的影响的度量标准。我们通过英语综合和真正的NLP任务加固我们的分析结果。

We introduce a new type of deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy). Our word vectors are learned functions of the internal states of a deep bidirectional language model (biLM), which is pretrained on a large text corpus. We show that these representations can be easily added to existing models and significantly improve the state of the art across six challenging NLP problems, including question answering, textual entailment and sentiment analysis. We also present an analysis showing that exposing the deep internals of the pre-trained network is crucial, allowing downstream models to mix different types of semi-supervision signals.

Pre-trained text encoders have rapidly advanced the state of the art on many NLP tasks. We focus on one such model, BERT, and aim to quantify where linguistic information is captured within the network. We find that the model represents the steps of the traditional NLP pipeline in an interpretable and localizable way, and that the regions responsible for each step appear in the expected sequence: POS tagging, parsing, NER, semantic roles, then coreference. Qualitative analysis reveals that the model can and often does adjust this pipeline dynamically, revising lowerlevel decisions on the basis of disambiguating information from higher-level representations.

We propose reconstruction probing, a new analysis method for contextualized representations based on reconstruction probabilities in masked language models (MLMs). This method relies on comparing the reconstruction probabilities of tokens in a given sequence when conditioned on the representation of a single token that has been fully contextualized and when conditioned on only the decontextualized lexical prior of the model. This comparison can be understood as quantifying the contribution of contextualization towards reconstruction -- the difference in the reconstruction probabilities can only be attributed to the representational change of the single token induced by contextualization. We apply this analysis to three MLMs and find that contextualization boosts reconstructability of tokens that are close to the token being reconstructed in terms of linear and syntactic distance. Furthermore, we extend our analysis to finer-grained decomposition of contextualized representations, and we find that these boosts are largely attributable to static and positional embeddings at the input layer.

对于自然语言处理系统，两种证据支持在大型未解除的基层上的神经语言模型中使用文本表示：在应用程序启发基准上的表现（Peters等，2018年，除其他外）以及出现的出现这些陈述中的句法抽象（Tenney等，2019年，尤其）。另一方面，缺乏接地的监督呼吁质疑这些表现如何捕获意义（Bender和Koller，2020）。我们对最近的语言模型应用小说探针 - 特别关注由语义依赖性运作的谓词参数结构（Ivanova等，2012） - 并发现，与语法不同，语义不是通过今天的预磨款模型带到表面上。然后，我们使用卷积图编码器将语义解析明确地将语义解析结合到特定于任务的FineTuning中，为胶水基准测试中的自然语言理解（NLU）任务产生益处。这种方法展示了通用（而不是任务特定的）语言监督的潜力，以上和超越传统的预威胁和芬特。有几个诊断有助于本地化我们方法的好处。

我们研究了原则上的程度，原则上，语言图表表示可以补充和改进神经语言建模。通过一个由7种不同的形式主义之一的预磨削变压器和地面真相图组成的集合设置，我们发现，总体而言，语义构成结构对语言建模性能最有用 - 超越句法选区结构以及句法和语义依赖结构。此外，效果取决于语音级别的级别大大变化。总而言之，我们的调查结果指出了神经象征性语言建模的有希望的趋势，并邀请未来的研究量化不同形式主义所制作的设计选择。

预先接受训练的语言模型的进展导致了对自然语言理解的下游任务的令人印象深刻的结果。探索预先训练的语言模型的最新工作揭示了在其上下围化表示中编码的广泛的语言属性。然而，目前尚不清楚他们是否编码对符号推理方法至关重要的语义知识。我们提出了一种用于探测预先接受训练的语言模型表示的逻辑推断的语言信息的方法。我们的探测数据集涵盖主要符号推理系统所需的语言现象列表。我们发现（i）预先接受的语言模型为推断编码几种类型的语言信息，但是还有一些类型的信息弱编码，（ii）语言模型可以通过微调有效地学习语言信息缺少语言信息。总体而言，我们的调查结果提供了逻辑推理语言模型的语言信息的洞察力，以及他们的预训练程序捕获。此外，我们已经证明了语言模型作为语义和背景知识库的潜力，用于支持符号推断方法。

Large pre-trained neural networks such as BERT have had great recent success in NLP, motivating a growing body of research investigating what aspects of language they are able to learn from unlabeled data. Most recent analysis has focused on model outputs (e.g., language model surprisal) or internal vector representations (e.g., probing classifiers). Complementary to these works, we propose methods for analyzing the attention mechanisms of pre-trained models and apply them to BERT. BERT's attention heads exhibit patterns such as attending to delimiter tokens, specific positional offsets, or broadly attending over the whole sentence, with heads in the same layer often exhibiting similar behaviors. We further show that certain attention heads correspond well to linguistic notions of syntax and coreference. For example, we find heads that attend to the direct objects of verbs, determiners of nouns, objects of prepositions, and coreferent mentions with remarkably high accuracy. Lastly, we propose an attention-based probing classifier and use it to further demonstrate that substantial syntactic information is captured in BERT's attention. 1 Code will be released at https://github.com/ clarkkev/attention-analysis.2 We use the English base-sized model.

多语言语言模型（\ mllms），如mbert，xlm，xlm-r，\ textit {etc。}已成为一种可行的选择，使预先估计到大量语言的力量。鉴于他们的成功在零射击转移学习中，在（i）建立更大的\ mllms〜覆盖了大量语言（ii）创建覆盖更广泛的任务和语言来评估的详尽工作基准mllms〜（iii）分析单音零点，零拍摄交叉和双语任务（iv）对Monolingual的性能，了解\ mllms〜（v）增强（通常）学习的通用语言模式（如果有的话）有限的容量\ mllms〜以提高他们在已见甚至看不见语言的表现。在这项调查中，我们审查了现有的文学，涵盖了上述与\ MLLMS有关的广泛研究领域。根据我们的调查，我们建议您有一些未来的研究方向。

尽管在理解深度NLP模型中学到的表示形式以及他们所捕获的知识方面已经做了很多工作，但对单个神经元的关注很少。我们提出了一种称为语言相关性分析的技术，可在任何外部特性中提取模型中的显着神经元 - 目的是了解如何保留这种知识在神经元中。我们进行了细粒度的分析以回答以下问题：（i）我们可以识别网络中捕获特定语言特性的神经元子集吗？ （ii）整个网络中的局部或分布式神经元如何？ iii）信息保留了多么冗余？ iv）针对下游NLP任务的微调预训练模型如何影响学习的语言知识？ iv）架构在学习不同的语言特性方面有何不同？我们的数据驱动的定量分析阐明了有趣的发现：（i）我们发现了可以预测不同语言任务的神经元的小亚集，ii）捕获基本的词汇信息（例如后缀），而这些神经元位于较低的大多数层中，iii，iii），而这些神经元，而那些神经元，而那些神经元则可以预测。学习复杂的概念（例如句法角色）主要是在中间和更高层中，iii），在转移学习过程中，显着的语言神经元从较高到较低的层移至较低的层，因为网络保留了较高的层以特定于任务信息，iv）我们发现很有趣在培训预训练模型之间的差异，关于如何保留语言信息，V）我们发现概念在多语言变压器模型中跨不同语言表现出相似的神经元分布。我们的代码作为Neurox工具包的一部分公开可用。

数据饥饿的深度神经网络已经将自己作为许多NLP任务的标准建立为包括传统序列标记的标准。尽管他们在高资源语言上表现最先进的表现，但它们仍然落后于低资源场景的统计计数器。一个方法来反击攻击此问题是文本增强，即，从现有数据生成新的合成训练数据点。虽然NLP最近目睹了一种文本增强技术的负载，但该领域仍然缺乏对多种语言和序列标记任务的系统性能分析。为了填补这一差距，我们调查了三类文本增强方法，其在语法（例如，裁剪子句子），令牌（例如，随机字插入）和字符（例如，字符交换）级别上执行更改。我们系统地将它们与语音标记，依赖解析和语义角色标记的分组进行了比较，用于使用各种模型的各种语言系列，包括依赖于诸如MBERT的普赖金的多语言语境化语言模型的架构。增强最显着改善了解析，然后是语音标记和语义角色标记的依赖性解析。我们发现实验技术通常在形态上丰富的语言，而不是越南语等分析语言。我们的研究结果表明，增强技术可以进一步改善基于MBERT的强基线。我们将字符级方法标识为最常见的表演者，而同义词替换和语法增强仪提供不一致的改进。最后，我们讨论了最大依赖于任务，语言对和模型类型的结果。

探测是一种流行的方法，可以辨别预先训练的语言模型表示中包含哪些语言信息。但是，选择探针模型的机制最近受到了激烈的争论，因为尚不清楚探针是否只是在提取信息或对语言属性进行建模。为了应对这一挑战，本文通过将探测作为提示任务提出探测来介绍一种新颖的探测方法。我们对五个探测任务进行实验，并表明我们的方法在提取信息方面比诊断探针更为可比或更好，而自行学习得更少。我们通过提示方法与注意力头修剪进一步结合探测，以分析模型将语言信息存储在其体系结构中的位置。然后，我们通过删除对该属性至关重要的头部并评估所得模型在语言建模上的性能来检查特定语言属性对预训练的有用性。

Deep Learning and Machine Learning based models have become extremely popular in text processing and information retrieval. However, the non-linear structures present inside the networks make these models largely inscrutable. A significant body of research has focused on increasing the transparency of these models. This article provides a broad overview of research on the explainability and interpretability of natural language processing and information retrieval methods. More specifically, we survey approaches that have been applied to explain word embeddings, sequence modeling, attention modules, transformers, BERT, and document ranking. The concluding section suggests some possible directions for future research on this topic.

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a;Radford et al., 2018), BERT is designed to pretrain deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be finetuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial taskspecific architecture modifications.BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

The rapid advancement of AI technology has made text generation tools like GPT-3 and ChatGPT increasingly accessible, scalable, and effective. This can pose serious threat to the credibility of various forms of media if these technologies are used for plagiarism, including scientific literature and news sources. Despite the development of automated methods for paraphrase identification, detecting this type of plagiarism remains a challenge due to the disparate nature of the datasets on which these methods are trained. In this study, we review traditional and current approaches to paraphrase identification and propose a refined typology of paraphrases. We also investigate how this typology is represented in popular datasets and how under-representation of certain types of paraphrases impacts detection capabilities. Finally, we outline new directions for future research and datasets in the pursuit of more effective paraphrase detection using AI.

Recent progress in pretraining language models on large textual corpora led to a surge of improvements for downstream NLP tasks. Whilst learning linguistic knowledge, these models may also be storing relational knowledge present in the training data, and may be able to answer queries structured as "fillin-the-blank" cloze statements. Language models have many advantages over structured knowledge bases: they require no schema engineering, allow practitioners to query about an open class of relations, are easy to extend to more data, and require no human supervision to train. We present an in-depth analysis of the relational knowledge already present (without fine-tuning) in a wide range of state-of-theart pretrained language models. We find that (i) without fine-tuning, BERT contains relational knowledge competitive with traditional NLP methods that have some access to oracle knowledge, (ii) BERT also does remarkably well on open-domain question answering against a supervised baseline, and (iii) certain types of factual knowledge are learned much more readily than others by standard language model pretraining approaches. The surprisingly strong ability of these models to recall factual knowledge without any fine-tuning demonstrates their potential as unsupervised open-domain QA systems. The code to reproduce our analysis is available at https: //github.com/facebookresearch/LAMA.