我们研究了原则上的程度，原则上，语言图表表示可以补充和改进神经语言建模。通过一个由7种不同的形式主义之一的预磨削变压器和地面真相图组成的集合设置，我们发现，总体而言，语义构成结构对语言建模性能最有用 - 超越句法选区结构以及句法和语义依赖结构。此外，效果取决于语音级别的级别大大变化。总而言之，我们的调查结果指出了神经象征性语言建模的有希望的趋势，并邀请未来的研究量化不同形式主义所制作的设计选择。

Contextualized representation models such as ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2018) have recently achieved state-of-the-art results on a diverse array of downstream NLP tasks. Building on recent token-level probing work, we introduce a novel edge probing task design and construct a broad suite of sub-sentence tasks derived from the traditional structured NLP pipeline. We probe word-level contextual representations from four recent models and investigate how they encode sentence structure across a range of syntactic, semantic, local, and long-range phenomena. We find that existing models trained on language modeling and translation produce strong representations for syntactic phenomena, but only offer comparably small improvements on semantic tasks over a non-contextual baseline.

基于变压器的语言模型最近在许多自然语言任务中取得了显着的结果。但是，通常通过利用大量培训数据来实现排行榜的性能，并且很少通过将明确的语言知识编码为神经模型。这使许多人质疑语言学对现代自然语言处理的相关性。在本文中，我介绍了几个案例研究，以说明理论语言学和神经语言模型仍然相互关联。首先，语言模型通过提供一个客观的工具来测量语义距离，这对语言学家很有用，语义距离很难使用传统方法。另一方面，语言理论通过提供框架和数据源来探究我们的语言模型，以了解语言理解的特定方面，从而有助于语言建模研究。本论文贡献了三项研究，探讨了语言模型中语法 - 听觉界面的不同方面。在论文的第一部分中，我将语言模型应用于单词类灵活性的问题。我将Mbert作为语义距离测量的来源，我提供了有利于将单词类灵活性分析为方向过程的证据。在论文的第二部分中，我提出了一种方法来测量语言模型中间层的惊奇方法。我的实验表明，包含形态句法异常的句子触发了语言模型早期的惊喜，而不是语义和常识异常。最后，在论文的第三部分中，我适应了一些心理语言学研究，以表明语言模型包含了论证结构结构的知识。总而言之，我的论文在自然语言处理，语言理论和心理语言学之间建立了新的联系，以为语言模型的解释提供新的观点。

Contextual word representations derived from large-scale neural language models are successful across a diverse set of NLP tasks, suggesting that they encode useful and transferable features of language. To shed light on the linguistic knowledge they capture, we study the representations produced by several recent pretrained contextualizers (variants of ELMo, the OpenAI transformer language model, and BERT) with a suite of seventeen diverse probing tasks. We find that linear models trained on top of frozen contextual representations are competitive with state-of-the-art task-specific models in many cases, but fail on tasks requiring fine-grained linguistic knowledge (e.g., conjunct identification). To investigate the transferability of contextual word representations, we quantify differences in the transferability of individual layers within contextualizers, especially between recurrent neural networks (RNNs) and transformers. For instance, higher layers of RNNs are more taskspecific, while transformer layers do not exhibit the same monotonic trend. In addition, to better understand what makes contextual word representations transferable, we compare language model pretraining with eleven supervised pretraining tasks. For any given task, pretraining on a closely related task yields better performance than language model pretraining (which is better on average) when the pretraining dataset is fixed. However, language model pretraining on more data gives the best results.

对于自然语言处理系统，两种证据支持在大型未解除的基层上的神经语言模型中使用文本表示：在应用程序启发基准上的表现（Peters等，2018年，除其他外）以及出现的出现这些陈述中的句法抽象（Tenney等，2019年，尤其）。另一方面，缺乏接地的监督呼吁质疑这些表现如何捕获意义（Bender和Koller，2020）。我们对最近的语言模型应用小说探针 - 特别关注由语义依赖性运作的谓词参数结构（Ivanova等，2012） - 并发现，与语法不同，语义不是通过今天的预磨款模型带到表面上。然后，我们使用卷积图编码器将语义解析明确地将语义解析结合到特定于任务的FineTuning中，为胶水基准测试中的自然语言理解（NLU）任务产生益处。这种方法展示了通用（而不是任务特定的）语言监督的潜力，以上和超越传统的预威胁和芬特。有几个诊断有助于本地化我们方法的好处。

在本文中，我们试图通过引入深度学习模型的句法归纳偏见来建立两所学校之间的联系。我们提出了两个归纳偏见的家族，一个家庭用于选区结构，另一个用于依赖性结构。选区归纳偏见鼓励深度学习模型使用不同的单位（或神经元）分别处理长期和短期信息。这种分离为深度学习模型提供了一种方法，可以从顺序输入中构建潜在的层次表示形式，即更高级别的表示由高级表示形式组成，并且可以分解为一系列低级表示。例如，在不了解地面实际结构的情况下，我们提出的模型学会通过根据其句法结构组成变量和运算符的表示来处理逻辑表达。另一方面，依赖归纳偏置鼓励模型在输入序列中找到实体之间的潜在关系。对于自然语言，潜在关系通常被建模为一个定向依赖图，其中一个单词恰好具有一个父节点和零或几个孩子的节点。将此约束应用于类似变压器的模型之后，我们发现该模型能够诱导接近人类专家注释的有向图，并且在不同任务上也优于标准变压器模型。我们认为，这些实验结果为深度学习模型的未来发展展示了一个有趣的选择。

Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue and approaches to compression. We then outline directions for future research.

We introduce Transformer Grammars (TGs), a novel class of Transformer language models that combine (i) the expressive power, scalability, and strong performance of Transformers and (ii) recursive syntactic compositions, which here are implemented through a special attention mask and deterministic transformation of the linearized tree. We find that TGs outperform various strong baselines on sentence-level language modeling perplexity, as well as on multiple syntax-sensitive language modeling evaluation metrics. Additionally, we find that the recursive syntactic composition bottleneck which represents each sentence as a single vector harms perplexity on document-level language modeling, providing evidence that a different kind of memory mechanism -- one that is independent of composed syntactic representations -- plays an important role in current successful models of long text.

在NLP社区中有一个正在进行的辩论，无论现代语言模型是否包含语言知识，通过所谓的探针恢复。在本文中，我们研究了语言知识是否是现代语言模型良好表现的必要条件，我们称之为\ Texit {重新发现假设}。首先，我们展示了语言模型，这是显着压缩的，但在预先磨普目标上表现良好，以便在语言结构探讨时保持良好的分数。这一结果支持重新发现的假设，并导致我们的论文的第二款贡献：一个信息 - 理论框架，与语言建模目标相关。该框架还提供了测量语言信息对字词预测任务的影响的度量标准。我们通过英语综合和真正的NLP任务加固我们的分析结果。

数据饥饿的深度神经网络已经将自己作为许多NLP任务的标准建立为包括传统序列标记的标准。尽管他们在高资源语言上表现最先进的表现，但它们仍然落后于低资源场景的统计计数器。一个方法来反击攻击此问题是文本增强，即，从现有数据生成新的合成训练数据点。虽然NLP最近目睹了一种文本增强技术的负载，但该领域仍然缺乏对多种语言和序列标记任务的系统性能分析。为了填补这一差距，我们调查了三类文本增强方法，其在语法（例如，裁剪子句子），令牌（例如，随机字插入）和字符（例如，字符交换）级别上执行更改。我们系统地将它们与语音标记，依赖解析和语义角色标记的分组进行了比较，用于使用各种模型的各种语言系列，包括依赖于诸如MBERT的普赖金的多语言语境化语言模型的架构。增强最显着改善了解析，然后是语音标记和语义角色标记的依赖性解析。我们发现实验技术通常在形态上丰富的语言，而不是越南语等分析语言。我们的研究结果表明，增强技术可以进一步改善基于MBERT的强基线。我们将字符级方法标识为最常见的表演者，而同义词替换和语法增强仪提供不一致的改进。最后，我们讨论了最大依赖于任务，语言对和模型类型的结果。

自然语言处理的机器学习快速进步有可能改变有关人类学习语言的辩论。但是，当前人工学习者和人类的学习环境和偏见以削弱从学习模拟获得的证据的影响的方式分歧。例如，当今最有效的神经语言模型接受了典型儿童可用的语言数据量的大约一千倍。为了增加计算模型的可学习性结果的相关性，我们需要培训模型学习者，而没有比人类具有显着优势的学习者。如果合适的模型成功地获得了一些目标语言知识，则可以提供一个概念证明，即在假设的人类学习方案中可以学习目标。合理的模型学习者将使我们能够进行实验操作，以对学习环境中的变量进行因果推断，并严格测试史密斯风格的贫困声明，主张根据人类对人类的先天语言知识，基于有关可学习性的猜测。由于实用和道德的考虑因素，人类受试者将永远无法实现可比的实验，从而使模型学习者成为必不可少的资源。到目前为止，试图剥夺当前模型的不公平优势，为关键语法行为（例如可接受性判断）获得亚人类结果。但是，在我们可以合理地得出结论，语言学习需要比当前模型拥有更多的特定领域知识，我们必须首先以多模式刺激和多代理互动的形式探索非语言意见，以使学习者更有效地学习学习者来自有限的语言输入。

我们想要模型的文本单位是什么？从字节到多字表达式，可以在许多粒度下分析和生成文本。直到最近，大多数自然语言处理（NLP）模型通过单词操作，将那些作为离散和原子令牌处理，但从字节对编码（BPE）开始，基于次字的方法在许多领域都变得占主导地位，使得仍然存在小词汇表允许快速推断。是道路字符级模型的结束或字节级处理吗？在这项调查中，我们通过展示和评估基于学习分割的词语和字符以及基于子字的方法的混合方法以及基于学习的分割的杂交方法，连接多行工作。我们得出结论，对于所有应用来说，并且可能永远不会成为所有应用的银子弹奇异解决方案，并且严重思考令牌化对许多应用仍然很重要。

我们研究了现代神经语言模型容易受到结构启动的程度，这种现象使句子的结构在后续句子中更有可能使相同的结构更有可能。我们探索如何使用启动来研究这些模型学习抽象结构信息的潜力，这是需要自然语言理解技能的任务良好表现的先决条件。我们引入了一种新型的度量标准和释放Prime-LM，这是一个大型语料库，我们可以控制与启动强度相互作用的各种语言因素。我们发现，变压器模型确实显示了结构启动的证据，但他们所学到的概括在某种程度上是由语义信息调节的。我们的实验还表明，模型获得的表示不仅可以编码抽象的顺序结构，而且还涉及一定级别的层次句法信息。更普遍的是，我们的研究表明，启动范式是一种有用的，可用于洞悉语言模型能力的有用的，并为未来的基于底漆的调查打开了探测模型内部状态的未来大门。

The rapid advancement of AI technology has made text generation tools like GPT-3 and ChatGPT increasingly accessible, scalable, and effective. This can pose serious threat to the credibility of various forms of media if these technologies are used for plagiarism, including scientific literature and news sources. Despite the development of automated methods for paraphrase identification, detecting this type of plagiarism remains a challenge due to the disparate nature of the datasets on which these methods are trained. In this study, we review traditional and current approaches to paraphrase identification and propose a refined typology of paraphrases. We also investigate how this typology is represented in popular datasets and how under-representation of certain types of paraphrases impacts detection capabilities. Finally, we outline new directions for future research and datasets in the pursuit of more effective paraphrase detection using AI.

当前的语言模型可以产生高质量的文本。他们只是复制他们之前看到的文本，或者他们学习了普遍的语言抽象吗？要取笑这些可能性，我们介绍了乌鸦，这是一套评估生成文本的新颖性，专注于顺序结构（n-gram）和句法结构。我们将这些分析应用于四种神经语言模型（LSTM，变压器，变换器-XL和GPT-2）。对于本地结构 - 例如，单个依赖性 - 模型生成的文本比来自每个模型的测试集的人类生成文本的基线显着不那么新颖。对于大规模结构 - 例如，总句结构 - 模型生成的文本与人生成的基线一样新颖甚至更新颖，但模型仍然有时复制，在某些情况下，在训练集中重复超过1000字超过1,000字的通道。我们还表现了广泛的手动分析，表明GPT-2的新文本通常在形态学和语法中形成良好，但具有合理的语义问题（例如，是自相矛盾）。

The long-distance agreement, evidence for syntactic structure, is increasingly used to assess the syntactic generalization of Neural Language Models. Much work has shown that transformers are capable of high accuracy in varied agreement tasks, but the mechanisms by which the models accomplish this behavior are still not well understood. To better understand transformers' internal working, this work contrasts how they handle two superficially similar but theoretically distinct agreement phenomena: subject-verb and object-past participle agreement in French. Using probing and counterfactual analysis methods, our experiments show that i) the agreement task suffers from several confounders which partially question the conclusions drawn so far and ii) transformers handle subject-verb and object-past participle agreements in a way that is consistent with their modeling in theoretical linguistics.

自然语言处理（NLP）已成为当前人工智能繁荣中的主要应用领域之一。转移学习已经启用了大量深入学习的神经网络，接受了语言建模任务，以大大提高了所有语言任务的性能。有趣的是，当模型培训使用包含软件代码的数据培训时，它们在从自然语言规范中生成功能计算机代码时展示了显着的能力。我们认为这是一种难题，用于神经模型为生成词组结构语法提供了一种替代理论，以说明语言有效。由于编程语言的语法由短语结构语法决定，因此成功的神经模型显然是对编程语言的理论基础的理论基础，以及通过扩展，自然语言来实现。我们认为语言模型的术语模型是误导性的，因为深度学习模型不是语言的理论模型，并提出采用语料库模型，这更好地反映了模型的成因和内容。

Recent progress in pretraining language models on large textual corpora led to a surge of improvements for downstream NLP tasks. Whilst learning linguistic knowledge, these models may also be storing relational knowledge present in the training data, and may be able to answer queries structured as "fillin-the-blank" cloze statements. Language models have many advantages over structured knowledge bases: they require no schema engineering, allow practitioners to query about an open class of relations, are easy to extend to more data, and require no human supervision to train. We present an in-depth analysis of the relational knowledge already present (without fine-tuning) in a wide range of state-of-theart pretrained language models. We find that (i) without fine-tuning, BERT contains relational knowledge competitive with traditional NLP methods that have some access to oracle knowledge, (ii) BERT also does remarkably well on open-domain question answering against a supervised baseline, and (iii) certain types of factual knowledge are learned much more readily than others by standard language model pretraining approaches. The surprisingly strong ability of these models to recall factual knowledge without any fine-tuning demonstrates their potential as unsupervised open-domain QA systems. The code to reproduce our analysis is available at https: //github.com/facebookresearch/LAMA.

Deep Learning and Machine Learning based models have become extremely popular in text processing and information retrieval. However, the non-linear structures present inside the networks make these models largely inscrutable. A significant body of research has focused on increasing the transparency of these models. This article provides a broad overview of research on the explainability and interpretability of natural language processing and information retrieval methods. More specifically, we survey approaches that have been applied to explain word embeddings, sequence modeling, attention modules, transformers, BERT, and document ranking. The concluding section suggests some possible directions for future research on this topic.

Syntax is a latent hierarchical structure which underpins the robust and compositional nature of human language. An active line of inquiry is whether large pretrained language models (LLMs) are able to acquire syntax by training on text alone; understanding a model's syntactic capabilities is essential to understanding how it processes and makes use of language. In this paper, we propose a new method, SSUD, which allows for the induction of syntactic structures without supervision from gold-standard parses. Instead, we seek to define formalism-agnostic, model-intrinsic syntactic parses by using a property of syntactic relations: syntactic substitutability. We demonstrate both quantitative and qualitative gains on dependency parsing tasks using SSUD, and induce syntactic structures which we hope provide clarity into LLMs and linguistic representations, alike.