Human linguistic capacity is often characterized by compositionality and the generalization it enables -- human learners can produce and comprehend novel complex expressions by composing known parts. Several benchmarks exploit distributional control across training and test to gauge compositional generalization, where certain lexical items only occur in limited contexts during training. While recent work using these benchmarks suggests that pretrained models achieve impressive generalization performance, we argue that exposure to pretraining data may break the aforementioned distributional control. Using the COGS benchmark of Kim and Linzen (2020), we test two modified evaluation setups that control for this issue: (1) substituting context-controlled lexical items with novel character sequences, and (2) substituting them with special tokens represented by novel embeddings. We find that both of these setups lead to lower generalization performance in T5 (Raffel et al., 2020), suggesting that previously reported results have been overestimated due to uncontrolled lexical exposure during pretraining. The performance degradation is more extreme with novel embeddings, and the degradation increases with the amount of pretraining data, highlighting an interesting case of inverse scaling.

基于变压器的语言模型最近在许多自然语言任务中取得了显着的结果。但是，通常通过利用大量培训数据来实现排行榜的性能，并且很少通过将明确的语言知识编码为神经模型。这使许多人质疑语言学对现代自然语言处理的相关性。在本文中，我介绍了几个案例研究，以说明理论语言学和神经语言模型仍然相互关联。首先，语言模型通过提供一个客观的工具来测量语义距离，这对语言学家很有用，语义距离很难使用传统方法。另一方面，语言理论通过提供框架和数据源来探究我们的语言模型，以了解语言理解的特定方面，从而有助于语言建模研究。本论文贡献了三项研究，探讨了语言模型中语法 - 听觉界面的不同方面。在论文的第一部分中，我将语言模型应用于单词类灵活性的问题。我将Mbert作为语义距离测量的来源，我提供了有利于将单词类灵活性分析为方向过程的证据。在论文的第二部分中，我提出了一种方法来测量语言模型中间层的惊奇方法。我的实验表明，包含形态句法异常的句子触发了语言模型早期的惊喜，而不是语义和常识异常。最后，在论文的第三部分中，我适应了一些心理语言学研究，以表明语言模型包含了论证结构结构的知识。总而言之，我的论文在自然语言处理，语言理论和心理语言学之间建立了新的联系，以为语言模型的解释提供新的观点。

当一个神经语言模型（LM）适于执行新任务时，任务的哪些方面预测了模型的最终性能？在NLP中，LM概括到个别示例的系统特征很好，但LM对新任务的系统的系统性方面并不理解。我们使用500个程序生成的序列建模任务构建的新基准测试，展示了LM适应性的特性和限制的大规模实证研究。这些任务组合了语言处理的核心方面，包括词汇语义，序列处理，记忆，逻辑推理和世界知识。使用TaskBench500，我们评估了三个适应性的方面，发现：（1）适应程序在他们记忆小型数据集的能力中急剧差异; （2）在任务类型的子集中，适应程序表现出对复杂任务的组成适应性; （3）未能匹配培训标签分布，在预测个别标签的内在难度中解释了不匹配。我们的实验表明，可以系统地描述和理解新的任务，如新示例的泛化，以及讨论可以使用新基准研究的适应性的其他方面的结论。

Large-scale pretraining instills large amounts of knowledge in deep neural networks. This, in turn, improves the generalization behavior of these models in downstream tasks. What exactly are the limits to the generalization benefits of large-scale pretraining? Here, we report observations from some simple experiments aimed at addressing this question in the context of two semantic parsing tasks involving natural language, SCAN and COGS. We show that language models pretrained exclusively with non-English corpora, or even with programming language corpora, significantly improve out-of-distribution generalization in these benchmarks, compared with models trained from scratch, even though both benchmarks are English-based. This demonstrates the surprisingly broad transferability of pretrained representations and knowledge. Pretraining with a large-scale protein sequence prediction task, on the other hand, mostly deteriorates the generalization performance in SCAN and COGS, suggesting that pretrained representations do not transfer universally and that there are constraints on the similarity between the pretraining and downstream domains for successful transfer. Finally, we show that larger models are harder to train from scratch and their generalization accuracy is lower when trained up to convergence on the relatively small SCAN and COGS datasets, but the benefits of large-scale pretraining become much clearer with larger models.

神经网络模型通常概括到不匹配的域或分布不符。在NLP中，特别是当预期模型概括为合作的模型，即熟悉词汇和建筑的新组合时，尤其产生这个问题。我们调查促进从一个组成任务转移到另一个组成任务的学习的学习陈述：模型的代表和任务特定层在预先驾驶任务上具有不同的培训，使得它们概括为需要合成性的不匹配分裂。我们将此方法应用于语义解析，使用三个非常不同的数据集，COG，地理信息集和扫描，作为FineTuning和目标任务交替使用。我们的方法显着改善了在目标任务的测试组上的基线上的组成概括，在微调期间被列出。消融研究表征了所提出的算法中主要步骤的效用，并支持我们的假设。

英国主导填字游戏，英国的主要填字游戏是推进寻求语义复杂，高度成分语言的NLP系统的有希望的目标。密码线索像流利的自然语言一样读，但是由两部分进行前列组成：一个定义和播放密码需要字符级操作。专家人类使用创造性的智能来解决隐秘，灵活地结合语言，世界和领域知识。在本文中，我们进行了两个主要贡献。首先，我们向NLP系统提供了一个充满挑战的新基准，以便在更具创造性，人类的方式处理组成语言的挑战新的基准。在显示三种非神经方法和T5之后，一种最先进的神经语言模型，无法实现良好的性能，我们提出了第二种主要贡献：一种新的课程方法，其中模型首先进行微调在相关任务，如解读单词。我们还介绍了一个具有挑战性的数据拆分，检查子字标记模型的元语言功能，并通过扰动线索的晶片部分来调查模型系统性，表明T5表现出与人类解决部分符合的行为策略。虽然我们的课程方法大大提高了T5基线，但我们最好的模型仍然无法概括为人类可以的程度。因此，隐秘填字游戏仍然是NLP系统的未解决挑战和未来创新的潜在来源。

自然语言处理的机器学习快速进步有可能改变有关人类学习语言的辩论。但是，当前人工学习者和人类的学习环境和偏见以削弱从学习模拟获得的证据的影响的方式分歧。例如，当今最有效的神经语言模型接受了典型儿童可用的语言数据量的大约一千倍。为了增加计算模型的可学习性结果的相关性，我们需要培训模型学习者，而没有比人类具有显着优势的学习者。如果合适的模型成功地获得了一些目标语言知识，则可以提供一个概念证明，即在假设的人类学习方案中可以学习目标。合理的模型学习者将使我们能够进行实验操作，以对学习环境中的变量进行因果推断，并严格测试史密斯风格的贫困声明，主张根据人类对人类的先天语言知识，基于有关可学习性的猜测。由于实用和道德的考虑因素，人类受试者将永远无法实现可比的实验，从而使模型学习者成为必不可少的资源。到目前为止，试图剥夺当前模型的不公平优势，为关键语法行为（例如可接受性判断）获得亚人类结果。但是，在我们可以合理地得出结论，语言学习需要比当前模型拥有更多的特定领域知识，我们必须首先以多模式刺激和多代理互动的形式探索非语言意见，以使学习者更有效地学习学习者来自有限的语言输入。

虽然神经语言模型往往对自然语言理解（NLU）任务进行令人惊讶的令人惊讶，但它们的优势和局限性仍然很差。因此，受控的合成任务是用于诊断模型行为的越来越重要的资源。在这项工作中，我们专注于讲故事的理解，是NLU系统的核心竞争力。然而，讲故事的主要综合资源是Babi基准，缺乏可控任务生成的这种系统机制。我们开发Dyna-Babi，一种动态框架，提供对Babi中的任务生成的细粒度控制。我们通过构建一个组成概括的三项新任务来展示我们的想法，这是来自原始基准的重要评估设置。我们测试了为BABI开发的专用模型以及最先进的预训练方法，发现这两种方法都解决了原始任务（> 99％的精度），并且在组成泛化设置中都没有成功地成功地成功，表示原始培训数据的局限性。我们探索了增加原始数据的方法，发现，尽管多样化培训数据比简单地增加数据集尺寸更有用，但它仍然不足以驾驶鲁棒成分泛化（具有<70％的复杂组合物的精度）。我们的结果强调了高度可控任务发生器通过模型和数据开发的良性循环创建强大的NLU系统的重要性。

标准预审进的语言模型可在子字代币序列上运行，而无需直接访问组成每个令牌字符串表示的字符。我们探究了预审前的语言模型的嵌入层，并表明模型在一个令人惊讶的程度上学习了整个单词和子字代币的内部字符组成，而没有看到字符和令牌。我们的结果表明，罗伯塔（Roberta）的嵌入层具有足够的信息，可以准确地阐明词汇的三分之一，并在所有令牌类型上达到高平均角色Ngram重叠。我们进一步测试了使用其他字符信息丰富子词模型是否可以改善语言建模，并观察到该方法具有几乎相同的学习曲线，作为训练而无需基于拼写的丰富。总体而言，我们的结果表明，语言建模目标激励模型隐式学习一些拼写概念，并且明确教授模型如何拼写的方式似乎并没有增强其在此类任务上的绩效。

关于人类阅读的研究长期以来一直记录在阅读行为表明特定于任务的效果，但是建立一个通用模型来预测人类在给定任务中将显示什么的通用模型。我们介绍了Neat，这是人类阅读中注意力分配的计算模型，基于人类阅读优化了一项任务中关注经济和成功之间的权衡。我们的模型是使用当代神经网络建模技术实施的，并对注意力分配的分配方式在不同任务中如何变化做出明确的测试预测。我们在一项针对阅读理解任务的两个版本的眼影研究中对此进行了测试，发现我们的模型成功说明了整个任务的阅读行为。因此，我们的工作提供了证据表明，任务效果可以建模为对任务需求的最佳适应。

Humans can understand and produce new utterances effortlessly, thanks to their compositional skills. Once a person learns the meaning of a new verb "dax," he or she can immediately understand the meaning of "dax twice" or "sing and dax." In this paper, we introduce the SCAN domain, consisting of a set of simple compositional navigation commands paired with the corresponding action sequences. We then test the zero-shot generalization capabilities of a variety of recurrent neural networks (RNNs) trained on SCAN with sequence-to-sequence methods. We find that RNNs can make successful zero-shot generalizations when the differences between training and test commands are small, so that they can apply "mix-and-match" strategies to solve the task. However, when generalization requires systematic compositional skills (as in the "dax" example above), RNNs fail spectacularly. We conclude with a proof-of-concept experiment in neural machine translation, suggesting that lack of systematicity might be partially responsible for neural networks' notorious training data thirst.

当前的语言模型可以产生高质量的文本。他们只是复制他们之前看到的文本，或者他们学习了普遍的语言抽象吗？要取笑这些可能性，我们介绍了乌鸦，这是一套评估生成文本的新颖性，专注于顺序结构（n-gram）和句法结构。我们将这些分析应用于四种神经语言模型（LSTM，变压器，变换器-XL和GPT-2）。对于本地结构 - 例如，单个依赖性 - 模型生成的文本比来自每个模型的测试集的人类生成文本的基线显着不那么新颖。对于大规模结构 - 例如，总句结构 - 模型生成的文本与人生成的基线一样新颖甚至更新颖，但模型仍然有时复制，在某些情况下，在训练集中重复超过1000字超过1,000字的通道。我们还表现了广泛的手动分析，表明GPT-2的新文本通常在形态学和语法中形成良好，但具有合理的语义问题（例如，是自相矛盾）。

Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue and approaches to compression. We then outline directions for future research.

Compositional generalization is a basic mechanism in human language learning, which current neural networks struggle with. A recently proposed Disentangled sequence-to-sequence model (Dangle) shows promising generalization capability by learning specialized encodings for each decoding step. We introduce two key modifications to this model which encourage more disentangled representations and improve its compute and memory efficiency, allowing us to tackle compositional generalization in a more realistic setting. Specifically, instead of adaptively re-encoding source keys and values at each time step, we disentangle their representations and only re-encode keys periodically, at some interval. Our new architecture leads to better generalization performance across existing tasks and datasets, and a new machine translation benchmark which we create by detecting naturally occurring compositional patterns in relation to a training set. We show this methodology better emulates real-world requirements than artificial challenges.

令牌化是预用语言模型（PLMS）的基础。用于中文PLMS的现有销量化方法通常将每个角色视为不可分割的令牌。然而，它们忽略了中文写字系统的独特特征，其中附加语言信息在字符级别下方，即在子字符级别。要利用此类信息，我们提出了子字符（Sub Const for Short）标记。具体地，我们首先通过基于其字形或发音将每个汉字转换为短序列来编码输入文本，然后根据具有子字标记化的编码文本构造词汇表。实验结果表明，Sub Colar标记与现有标记均具有两个主要优点：1）它们可以将输入牌销料到更短的序列中，从而提高计算效率。 2）基于发音的Sub Col.Tokenizers可以将中文同音铭器编码为相同的音译序列并产生相同的标记输出，因此对所有同音声音拼写的强大。与此同时，使用Sub Colar标记培训的模型竞争地执行下游任务。我们在https://github.com/thunlp/subchartoken中发布我们的代码，以促进未来的工作。

People constantly use language to learn about the world. Computational linguists have capitalized on this fact to build large language models (LLMs) that acquire co-occurrence-based knowledge from language corpora. LLMs achieve impressive performance on many tasks, but the robustness of their world knowledge has been questioned. Here, we ask: do LLMs acquire generalized knowledge about real-world events? Using curated sets of minimal sentence pairs (n=1215), we tested whether LLMs are more likely to generate plausible event descriptions compared to their implausible counterparts. We found that LLMs systematically distinguish possible and impossible events (The teacher bought the laptop vs. The laptop bought the teacher) but fall short of human performance when distinguishing likely and unlikely events (The nanny tutored the boy vs. The boy tutored the nanny). In follow-up analyses, we show that (i) LLM scores are driven by both plausibility and surface-level sentence features, (ii) LLMs generalize well across syntactic sentence variants (active vs passive) but less well across semantic sentence variants (synonymous sentences), (iii) some, but not all LLM deviations from ground-truth labels align with crowdsourced human judgments, and (iv) explicit event plausibility information emerges in middle LLM layers and remains high thereafter. Overall, our analyses reveal a gap in LLMs' event knowledge, highlighting their limitations as generalized knowledge bases. We conclude by speculating that the differential performance on impossible vs. unlikely events is not a temporary setback but an inherent property of LLMs, reflecting a fundamental difference between linguistic knowledge and world knowledge in intelligent systems.

Recent datasets expose the lack of the systematic generalization ability in standard sequence-to-sequence models. In this work, we analyze this behavior of seq2seq models and identify two contributing factors: a lack of mutual exclusivity bias (i.e., a source sequence already mapped to a target sequence is less likely to be mapped to other target sequences), and the tendency to memorize whole examples rather than separating structures from contents. We propose two techniques to address these two issues respectively: Mutual Exclusivity Training that prevents the model from producing seen generations when facing novel, unseen examples via an unlikelihood-based loss; and prim2primX data augmentation that automatically diversifies the arguments of every syntactic function to prevent memorizing and provide a compositional inductive bias without exposing test-set data. Combining these two techniques, we show substantial empirical improvements using standard sequence-to-sequence models (LSTMs and Transformers) on two widely-used compositionality datasets: SCAN and COGS. Finally, we provide analysis characterizing the improvements as well as the remaining challenges, and provide detailed ablations of our method. Our code is available at https://github.com/owenzx/met-primaug

已显示通用非结构化神经网络在分布外的组成概述上挣扎。通过示例重组的组成数据增强已经转移了一些关于组成性的关于多个语义解析任务的黑盒神经模型的先前知识，但这通常需要特定于任务的工程或提供有限的收益。我们使用称为组成结构学习者（CSL）的型号提供更强大的数据重组方法。 CSL是一种具有拟同步无线语法骨干的生成模型，我们从训练数据中诱导。我们从CSL中进行重组的例子，并将其添加到预先训练的序列到序列模型（T5）的微调数据中。该程序有效地将大多数CSL的组成偏差转移到T5以进行诊断任务，并导致模型比在两个真实世界的组成泛化任务上的T5-CSL集合更强。这导致新的最先进的性能，这些挑战性的语义解析任务需要泛化自然语言变异和元素的新组成。

我们想要模型的文本单位是什么？从字节到多字表达式，可以在许多粒度下分析和生成文本。直到最近，大多数自然语言处理（NLP）模型通过单词操作，将那些作为离散和原子令牌处理，但从字节对编码（BPE）开始，基于次字的方法在许多领域都变得占主导地位，使得仍然存在小词汇表允许快速推断。是道路字符级模型的结束或字节级处理吗？在这项调查中，我们通过展示和评估基于学习分割的词语和字符以及基于子字的方法的混合方法以及基于学习的分割的杂交方法，连接多行工作。我们得出结论，对于所有应用来说，并且可能永远不会成为所有应用的银子弹奇异解决方案，并且严重思考令牌化对许多应用仍然很重要。

For natural language understanding (NLU) technology to be maximally useful, it must be able to process language in a way that is not exclusive to a single task, genre, or dataset. In pursuit of this objective, we introduce the General Language Understanding Evaluation (GLUE) benchmark, a collection of tools for evaluating the performance of models across a diverse set of existing NLU tasks. By including tasks with limited training data, GLUE is designed to favor and encourage models that share general linguistic knowledge across tasks. GLUE also includes a hand-crafted diagnostic test suite that enables detailed linguistic analysis of models. We evaluate baselines based on current methods for transfer and representation learning and find that multi-task training on all tasks performs better than training a separate model per task. However, the low absolute performance of our best model indicates the need for improved general NLU systems.