Targeted syntactic evaluations of language models ask whether models show stable preferences for syntactically acceptable content over minimal-pair unacceptable inputs. Most targeted syntactic evaluation datasets ask models to make these judgements with just a single context-free sentence as input. This does not match language models' training regime, in which input sentences are always highly contextualized by the surrounding corpus. This mismatch raises an important question: how robust are models' syntactic judgements in different contexts? In this paper, we investigate the stability of language models' performance on targeted syntactic evaluations as we vary properties of the input context: the length of the context, the types of syntactic phenomena it contains, and whether or not there are violations of grammaticality. We find that model judgements are generally robust when placed in randomly sampled linguistic contexts. However, they are substantially unstable for contexts containing syntactic structures matching those in the critical test content. Among all tested models (GPT-2 and five variants of OPT), we significantly improve models' judgements by providing contexts with matching syntactic structures, and conversely significantly worsen them using unacceptable contexts with matching but violated syntactic structures. This effect is amplified by the length of the context, except for unrelated inputs. We show that these changes in model performance are not explainable by simple features matching the context and the test inputs, such as lexical overlap and dependency overlap. This sensitivity to highly specific syntactic features of the context can only be explained by the models' implicit in-context learning abilities.

我们研究了现代神经语言模型容易受到结构启动的程度，这种现象使句子的结构在后续句子中更有可能使相同的结构更有可能。我们探索如何使用启动来研究这些模型学习抽象结构信息的潜力，这是需要自然语言理解技能的任务良好表现的先决条件。我们引入了一种新型的度量标准和释放Prime-LM，这是一个大型语料库，我们可以控制与启动强度相互作用的各种语言因素。我们发现，变压器模型确实显示了结构启动的证据，但他们所学到的概括在某种程度上是由语义信息调节的。我们的实验还表明，模型获得的表示不仅可以编码抽象的顺序结构，而且还涉及一定级别的层次句法信息。更普遍的是，我们的研究表明，启动范式是一种有用的，可用于洞悉语言模型能力的有用的，并为未来的基于底漆的调查打开了探测模型内部状态的未来大门。

基于变压器的语言模型最近在许多自然语言任务中取得了显着的结果。但是，通常通过利用大量培训数据来实现排行榜的性能，并且很少通过将明确的语言知识编码为神经模型。这使许多人质疑语言学对现代自然语言处理的相关性。在本文中，我介绍了几个案例研究，以说明理论语言学和神经语言模型仍然相互关联。首先，语言模型通过提供一个客观的工具来测量语义距离，这对语言学家很有用，语义距离很难使用传统方法。另一方面，语言理论通过提供框架和数据源来探究我们的语言模型，以了解语言理解的特定方面，从而有助于语言建模研究。本论文贡献了三项研究，探讨了语言模型中语法 - 听觉界面的不同方面。在论文的第一部分中，我将语言模型应用于单词类灵活性的问题。我将Mbert作为语义距离测量的来源，我提供了有利于将单词类灵活性分析为方向过程的证据。在论文的第二部分中，我提出了一种方法来测量语言模型中间层的惊奇方法。我的实验表明，包含形态句法异常的句子触发了语言模型早期的惊喜，而不是语义和常识异常。最后，在论文的第三部分中，我适应了一些心理语言学研究，以表明语言模型包含了论证结构结构的知识。总而言之，我的论文在自然语言处理，语言理论和心理语言学之间建立了新的联系，以为语言模型的解释提供新的观点。

当前的语言模型可以产生高质量的文本。他们只是复制他们之前看到的文本，或者他们学习了普遍的语言抽象吗？要取笑这些可能性，我们介绍了乌鸦，这是一套评估生成文本的新颖性，专注于顺序结构（n-gram）和句法结构。我们将这些分析应用于四种神经语言模型（LSTM，变压器，变换器-XL和GPT-2）。对于本地结构 - 例如，单个依赖性 - 模型生成的文本比来自每个模型的测试集的人类生成文本的基线显着不那么新颖。对于大规模结构 - 例如，总句结构 - 模型生成的文本与人生成的基线一样新颖甚至更新颖，但模型仍然有时复制，在某些情况下，在训练集中重复超过1000字超过1,000字的通道。我们还表现了广泛的手动分析，表明GPT-2的新文本通常在形态学和语法中形成良好，但具有合理的语义问题（例如，是自相矛盾）。

The long-distance agreement, evidence for syntactic structure, is increasingly used to assess the syntactic generalization of Neural Language Models. Much work has shown that transformers are capable of high accuracy in varied agreement tasks, but the mechanisms by which the models accomplish this behavior are still not well understood. To better understand transformers' internal working, this work contrasts how they handle two superficially similar but theoretically distinct agreement phenomena: subject-verb and object-past participle agreement in French. Using probing and counterfactual analysis methods, our experiments show that i) the agreement task suffers from several confounders which partially question the conclusions drawn so far and ii) transformers handle subject-verb and object-past participle agreements in a way that is consistent with their modeling in theoretical linguistics.

自然语言处理的机器学习快速进步有可能改变有关人类学习语言的辩论。但是，当前人工学习者和人类的学习环境和偏见以削弱从学习模拟获得的证据的影响的方式分歧。例如，当今最有效的神经语言模型接受了典型儿童可用的语言数据量的大约一千倍。为了增加计算模型的可学习性结果的相关性，我们需要培训模型学习者，而没有比人类具有显着优势的学习者。如果合适的模型成功地获得了一些目标语言知识，则可以提供一个概念证明，即在假设的人类学习方案中可以学习目标。合理的模型学习者将使我们能够进行实验操作，以对学习环境中的变量进行因果推断，并严格测试史密斯风格的贫困声明，主张根据人类对人类的先天语言知识，基于有关可学习性的猜测。由于实用和道德的考虑因素，人类受试者将永远无法实现可比的实验，从而使模型学习者成为必不可少的资源。到目前为止，试图剥夺当前模型的不公平优势，为关键语法行为（例如可接受性判断）获得亚人类结果。但是，在我们可以合理地得出结论，语言学习需要比当前模型拥有更多的特定领域知识，我们必须首先以多模式刺激和多代理互动的形式探索非语言意见，以使学习者更有效地学习学习者来自有限的语言输入。

People constantly use language to learn about the world. Computational linguists have capitalized on this fact to build large language models (LLMs) that acquire co-occurrence-based knowledge from language corpora. LLMs achieve impressive performance on many tasks, but the robustness of their world knowledge has been questioned. Here, we ask: do LLMs acquire generalized knowledge about real-world events? Using curated sets of minimal sentence pairs (n=1215), we tested whether LLMs are more likely to generate plausible event descriptions compared to their implausible counterparts. We found that LLMs systematically distinguish possible and impossible events (The teacher bought the laptop vs. The laptop bought the teacher) but fall short of human performance when distinguishing likely and unlikely events (The nanny tutored the boy vs. The boy tutored the nanny). In follow-up analyses, we show that (i) LLM scores are driven by both plausibility and surface-level sentence features, (ii) LLMs generalize well across syntactic sentence variants (active vs passive) but less well across semantic sentence variants (synonymous sentences), (iii) some, but not all LLM deviations from ground-truth labels align with crowdsourced human judgments, and (iv) explicit event plausibility information emerges in middle LLM layers and remains high thereafter. Overall, our analyses reveal a gap in LLMs' event knowledge, highlighting their limitations as generalized knowledge bases. We conclude by speculating that the differential performance on impossible vs. unlikely events is not a temporary setback but an inherent property of LLMs, reflecting a fundamental difference between linguistic knowledge and world knowledge in intelligent systems.

This work presents a detailed linguistic analysis into why larger Transformer-based pre-trained language models with more parameters and lower perplexity nonetheless yield surprisal estimates that are less predictive of human reading times. First, regression analyses show a strictly monotonic, positive log-linear relationship between perplexity and fit to reading times for the more recently released five GPT-Neo variants and eight OPT variants on two separate datasets, replicating earlier results limited to just GPT-2 (Oh et al., 2022). Subsequently, analysis of residual errors reveals a systematic deviation of the larger variants, such as underpredicting reading times of named entities and making compensatory overpredictions for reading times of function words such as modals and conjunctions. These results suggest that the propensity of larger Transformer-based models to 'memorize' sequences during training makes their surprisal estimates diverge from humanlike expectations, which warrants caution in using pre-trained language models to study human language processing.

鉴于大型语言模型的广泛能力，应该有可能朝着一般的文本的助手工作，这些助手与人类价值一致，这意味着它是有帮助，诚实的和无害的。在此方向上的初始遗传，我们研究简单的基线技术和评估，例如提示。我们发现，从模型规模增加适度的干预措施的好处，概括为各种对准评估，并不会损害大型模型的性能。接下来，我们调查与对齐，比较仿制，二进制歧视和排名偏好建模相关的几个培训目标的缩放趋势。我们发现排名优先级模型比模仿学习更好地表现得多，并且通常以模型大小更有利地缩放。相比之下，二进制歧视通常与模仿学习非常类似地执行和缩放。最后，我们研究了一种“偏好模型预训练阶段的培训阶段，其目的是在对人偏好的芬明时提高样本效率。

This paper investigates the ability of artificial neural networks to judge the grammatical acceptability of a sentence, with the goal of testing their linguistic competence. We introduce the Corpus of Linguistic Acceptability (CoLA), a set of 10,657 English sentences labeled as grammatical or ungrammatical from published linguistics literature. As baselines, we train several recurrent neural network models on acceptability classification, and find that our models outperform unsupervised models by Lau et al. (2016) on CoLA. Error-analysis on specific grammatical phenomena reveals that both Lau et al.'s models and ours learn systematic generalizations like subject-verb-object order. However, all models we test perform far below human level on a wide range of grammatical constructions.

GPT-3等大型自回归语言模型是几秒钟的学习者，可以在没有微调的情况下执行各种语言任务。虽然已知这些模型能够共同代表许多不同的语言，但他们的培训数据由英语主导，可能限制了它们的交叉概括。在这项工作中，我们在覆盖多种语言的平衡语料库上培训多语言自回归语言模型，并在广泛的任务中研究他们几乎没有零点的学习能力。我们最大的模型，具有75亿参数，在20多种代表语言中，在几种代表语言中，在几种代表性语言中，在几种代表性语言中，在多语言型号推理中表现出可比大小的GPT-3（在0次设置和0次拍摄设置中的绝对精度改善+ 7.4％ 4-拍摄设置中的9.4％）和自然语言推理（每次拍摄和4次设置中的每一个+ 5.4％）。在Flores-101机器翻译基准测试中，我们的模型优于GPT-3在182个翻译方向上有32个培训例子，同时超过45个方向的官方监督基线。我们介绍了模型成功和失败的位置的详细分析，特别是它尤其显示在某些任务中实现交叉语境的内容学习，而仍然存在改善表面的鲁棒性和适应没有a的任务的余地自然冻结形式。最后，我们评估我们在仇恨语音检测中以五种语言的仇恨语音检测的模型，并发现它具有与可比大小的GPT-3模型类似的限制。

We introduce Transformer Grammars (TGs), a novel class of Transformer language models that combine (i) the expressive power, scalability, and strong performance of Transformers and (ii) recursive syntactic compositions, which here are implemented through a special attention mask and deterministic transformation of the linearized tree. We find that TGs outperform various strong baselines on sentence-level language modeling perplexity, as well as on multiple syntax-sensitive language modeling evaluation metrics. Additionally, we find that the recursive syntactic composition bottleneck which represents each sentence as a single vector harms perplexity on document-level language modeling, providing evidence that a different kind of memory mechanism -- one that is independent of composed syntactic representations -- plays an important role in current successful models of long text.

在本文中，我们试图通过引入深度学习模型的句法归纳偏见来建立两所学校之间的联系。我们提出了两个归纳偏见的家族，一个家庭用于选区结构，另一个用于依赖性结构。选区归纳偏见鼓励深度学习模型使用不同的单位（或神经元）分别处理长期和短期信息。这种分离为深度学习模型提供了一种方法，可以从顺序输入中构建潜在的层次表示形式，即更高级别的表示由高级表示形式组成，并且可以分解为一系列低级表示。例如，在不了解地面实际结构的情况下，我们提出的模型学会通过根据其句法结构组成变量和运算符的表示来处理逻辑表达。另一方面，依赖归纳偏置鼓励模型在输入序列中找到实体之间的潜在关系。对于自然语言，潜在关系通常被建模为一个定向依赖图，其中一个单词恰好具有一个父节点和零或几个孩子的节点。将此约束应用于类似变压器的模型之后，我们发现该模型能够诱导接近人类专家注释的有向图，并且在不同任务上也优于标准变压器模型。我们认为，这些实验结果为深度学习模型的未来发展展示了一个有趣的选择。

抽象推理是智能系统的关键能力。大型语言模型在抽象推理任务上实现了高度的性能，但表现出许多缺陷。但是，人类的抽象推理也是不完美的，并且取决于我们对推理问题内容的知识和信念。例如，人类对在日常情况下基于逻辑规则的逻辑规则比关于抽象属性的任意规则更可靠地理解。语言模型的培训经验类似地赋予了他们先前的期望，这些期望反映了人类的知识和信念。因此，我们假设语言模型会显示出类似人类的内容对抽象推理问题的影响。我们在三个逻辑推理任务中探讨了这一假设：自然语言推论，判断三段论的逻辑有效性和ison选择任务（Wason，1968）。我们发现，最新的大语言模型（具有7或700亿个参数； Hoffman等，2022）反映了这些任务中人类在人类中观察到的许多相同模式 - 像人类一样，模型对可信情况的理由更有效地理由不现实或抽象的。我们的发现对理解这些认知效应以及有助于语言模型表现的因素具有影响。

Human evaluation is the foundation upon which the evaluation of both summarization systems and automatic metrics rests. However, existing human evaluation protocols and benchmarks for summarization either exhibit low inter-annotator agreement or lack the scale needed to draw statistically significant conclusions, and an in-depth analysis of human evaluation is lacking. In this work, we address the shortcomings of existing summarization evaluation along the following axes: 1) We propose a modified summarization salience protocol, Atomic Content Units (ACUs), which relies on fine-grained semantic units and allows for high inter-annotator agreement. 2) We curate the Robust Summarization Evaluation (RoSE) benchmark, a large human evaluation dataset consisting of over 22k summary-level annotations over state-of-the-art systems on three datasets. 3) We compare our ACU protocol with three other human evaluation protocols, underscoring potential confounding factors in evaluation setups. 4) We evaluate existing automatic metrics using the collected human annotations across evaluation protocols and demonstrate how our benchmark leads to more statistically stable and significant results. Furthermore, our findings have important implications for evaluating large language models (LLMs), as we show that LLMs adjusted by human feedback (e.g., GPT-3.5) may overfit unconstrained human evaluation, which is affected by the annotators' prior, input-agnostic preferences, calling for more robust, targeted evaluation methods.

Pragmatics is an essential part of communication, but it remains unclear what mechanisms underlie human pragmatic communication and whether NLP systems capture pragmatic language understanding. To investigate both these questions, we perform a fine-grained comparison of language models and humans on seven pragmatic phenomena, using zero-shot prompting on an expert-curated set of English materials. We ask whether models (1) select pragmatic interpretations of speaker utterances, (2) make similar error patterns as humans, and (3) use similar linguistic cues as humans to solve the tasks. We find that the largest models achieve high accuracy and match human error patterns: within incorrect responses, models favor the literal interpretation of an utterance over heuristic-based distractors. We also find evidence that models and humans are sensitive to similar linguistic cues. Our results suggest that even paradigmatic pragmatic phenomena may be solved without explicit representations of other agents' mental states, and that artificial models can be used to gain mechanistic insights into human pragmatic processing.

关于人类阅读的研究长期以来一直记录在阅读行为表明特定于任务的效果，但是建立一个通用模型来预测人类在给定任务中将显示什么的通用模型。我们介绍了Neat，这是人类阅读中注意力分配的计算模型，基于人类阅读优化了一项任务中关注经济和成功之间的权衡。我们的模型是使用当代神经网络建模技术实施的，并对注意力分配的分配方式在不同任务中如何变化做出明确的测试预测。我们在一项针对阅读理解任务的两个版本的眼影研究中对此进行了测试，发现我们的模型成功说明了整个任务的阅读行为。因此，我们的工作提供了证据表明，任务效果可以建模为对任务需求的最佳适应。

语言模型中的上下文化单词嵌入已为NLP提供了很大的进步。直观地，句子信息集成到单词的表示中，这可以帮助模型多义。但是，上下文灵敏度也导致表示形式的差异，这可能会破坏同义词的语义一致性。我们量化了典型的预训练模型中每个单词sense的上下文嵌入的程度各不相同。结果表明，在上下文中，上下文化的嵌入可以高度一致。此外，词性，单词感官的数量和句子长度对感官表示的差异有影响。有趣的是，我们发现单词表示是偏见的，在不同上下文中的第一个单词往往更相似。我们分析了这种现象，还提出了一种简单的方法来减轻基于距离的单词sense剥夺歧义设置的偏见。

我们研究了原则上的程度，原则上，语言图表表示可以补充和改进神经语言建模。通过一个由7种不同的形式主义之一的预磨削变压器和地面真相图组成的集合设置，我们发现，总体而言，语义构成结构对语言建模性能最有用 - 超越句法选区结构以及句法和语义依赖结构。此外，效果取决于语音级别的级别大大变化。总而言之，我们的调查结果指出了神经象征性语言建模的有希望的趋势，并邀请未来的研究量化不同形式主义所制作的设计选择。

For natural language understanding (NLU) technology to be maximally useful, it must be able to process language in a way that is not exclusive to a single task, genre, or dataset. In pursuit of this objective, we introduce the General Language Understanding Evaluation (GLUE) benchmark, a collection of tools for evaluating the performance of models across a diverse set of existing NLU tasks. By including tasks with limited training data, GLUE is designed to favor and encourage models that share general linguistic knowledge across tasks. GLUE also includes a hand-crafted diagnostic test suite that enables detailed linguistic analysis of models. We evaluate baselines based on current methods for transfer and representation learning and find that multi-task training on all tasks performs better than training a separate model per task. However, the low absolute performance of our best model indicates the need for improved general NLU systems.