Standard accuracy metrics indicate that reading comprehension systems are making rapid progress, but the extent to which these systems truly understand language remains unclear.To reward systems with real language understanding abilities, we propose an adversarial evaluation scheme for the Stanford Question Answering Dataset (SQuAD). Our method tests whether systems can answer questions about paragraphs that contain adversarially inserted sentences, which are automatically generated to distract computer systems without changing the correct answer or misleading humans. In this adversarial setting, the accuracy of sixteen published models drops from an average of 75% F1 score to 36%; when the adversary is allowed to add ungrammatical sequences of words, average accuracy on four models decreases further to 7%. We hope our insights will motivate the development of new models that understand language more precisely. Article: Super Bowl 50 Paragraph: "Peyton Manning became the first quarterback ever to lead two different teams to multiple Super Bowls. He is also the oldest quarterback ever to play in a Super Bowl at age 39. The past record was held by John Elway, who led the Broncos to victory in Super Bowl XXXIII at age 38 and is currently Denver's Executive Vice President of Football Operations and General Manager. Quarterback Jeff Dean had jersey number 37 in Champ Bowl XXXIV." Question: "What is the name of the quarterback who was 38 in Super Bowl XXXIII?"

Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for which the correct answer is not stated in the context. Existing datasets either focus exclusively on answerable questions, or use automatically generated unanswerable questions that are easy to identify. To address these weaknesses, we present SQuAD 2.0, the latest version of the Stanford Question Answering Dataset (SQuAD). SQuAD 2.0 combines existing SQuAD data with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. To do well on SQuAD 2.0, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering. SQuAD 2.0 is a challenging natural language understanding task for existing models: a strong neural system that gets 86% F1 on SQuAD 1.1 achieves only 66% F1 on SQuAD 2.0.

We present the Stanford Question Answering Dataset (SQuAD), a new reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage. We analyze the dataset to understand the types of reasoning required to answer the questions, leaning heavily on dependency and constituency trees. We build a strong logistic regression model, which achieves an F1 score of 51.0%, a significant improvement over a simple baseline (20%). However, human performance (86.8%) is much higher, indicating that the dataset presents a good challenge problem for future research. The dataset is freely available at https://stanford-qa.com.

关于NLP模型的最先进攻击缺乏对成功攻击的共享定义。我们将思考从过去的工作蒸馏成统一的框架：一个成功的自然语言对抗性示例是欺骗模型并遵循一些语言限制的扰动。然后，我们分析了两个最先进的同义词替换攻击的产出。我们发现他们的扰动通常不会保留语义，38％引入语法错误。人类调查显示，为了成功保留语义，我们需要大大增加交换词语的嵌入和原始和扰动句子的句子编码之间的最小余弦相似之处。与更好的保留语义和语法性，攻击成功率下降超过70个百分点。

Machine learning algorithms are often vulnerable to adversarial examples that have imperceptible alterations from the original counterparts but can fool the state-of-the-art models. It is helpful to evaluate or even improve the robustness of these models by exposing the maliciously crafted adversarial examples. In this paper, we present TEXTFOOLER, a simple but strong baseline to generate adversarial text. By applying it to two fundamental natural language tasks, text classification and textual entailment, we successfully attacked three target models, including the powerful pre-trained BERT, and the widely used convolutional and recurrent neural networks. We demonstrate three advantages of this framework:(1) effective-it outperforms previous attacks by success rate and perturbation rate, (2) utility-preserving-it preserves semantic content, grammaticality, and correct types classified by humans, and (3) efficient-it generates adversarial text with computational complexity linear to the text length. 1

Humans gather information through conversations involving a series of interconnected questions and answers. For machines to assist in information gathering, it is therefore essential to enable them to answer conversational questions. We introduce CoQA, a novel dataset for building Conversational Question Answering systems. 1 Our dataset contains 127k questions with answers, obtained from 8k conversations about text passages from seven diverse domains. The questions are conversational, and the answers are free-form text with their corresponding evidence highlighted in the passage. We analyze CoQA in depth and show that conversational questions have challenging phenomena not present in existing reading comprehension datasets, e.g., coreference and pragmatic reasoning. We evaluate strong dialogue and reading comprehension models on CoQA. The best system obtains an F1 score of 65.4%, which is 23.4 points behind human performance (88.8%), indicating there is ample room for improvement. We present CoQA as a challenge to the community at https://stanfordnlp. github.io/coqa.

近年来，在挑战的多跳QA任务方面有令人印象深刻的进步。然而，当面对输入文本中的一些干扰时，这些QA模型可能会失败，并且它们进行多跳推理的可解释性仍然不确定。以前的逆势攻击作品通常编辑整个问题句，这对测试基于实体的多跳推理能力有限。在本文中，我们提出了一种基于多跳推理链的逆势攻击方法。我们将从查询实体开始的多跳推理链与构造的图表中的答案实体一起制定，这使我们能够将问题对齐到每个推理跳跃，从而攻击任何跃点。我们将问题分类为不同的推理类型和对应于所选推理跳的部分问题，以产生分散注意力的句子。我们在HotpotQA DataSet上的三个QA模型上测试我们的对抗方案。结果表明，对答案和支持事实预测的显着性能降低，验证了我们推理基于链条推理模型的攻击方法的有效性以及它们的脆弱性。我们的对抗重新培训进一步提高了这些模型的性能和鲁棒性。

Winograd架构挑战 - 一套涉及代词参考消歧的双句话，似乎需要使用致辞知识 - 是由2011年的赫克托勒维克斯提出的。到2019年，基于大型预先训练的变压器的一些AI系统基于语言模型和微调这些问题，精度优于90％。在本文中，我们审查了Winograd架构挑战的历史并评估了其重要性。

在宣传，新闻和社交媒体中的虚假，不准确和误导信息中，现实世界的问题应答（QA）系统面临综合和推理相互矛盾的挑战，以获得正确答案的挑战。这种紧迫性导致需要使QA系统对错误信息的强大，这是一个先前未开发的主题。我们通过调查与实际和虚假信息混合的矛盾的情况下，通过调查QA模型的行为来研究对QA模型的错误信息的风险。我们为此问题创建了第一个大规模数据集，即对QA，其中包含超过10K的人写和模型生成的矛盾的上下文。实验表明，QA模型易受误导的背景下的攻击。为了防御这种威胁，我们建立一个错误信息感知的QA系统作为一个反措施，可以以联合方式整合问题应答和错误信息检测。

当前的语言模型可以产生高质量的文本。他们只是复制他们之前看到的文本，或者他们学习了普遍的语言抽象吗？要取笑这些可能性，我们介绍了乌鸦，这是一套评估生成文本的新颖性，专注于顺序结构（n-gram）和句法结构。我们将这些分析应用于四种神经语言模型（LSTM，变压器，变换器-XL和GPT-2）。对于本地结构 - 例如，单个依赖性 - 模型生成的文本比来自每个模型的测试集的人类生成文本的基线显着不那么新颖。对于大规模结构 - 例如，总句结构 - 模型生成的文本与人生成的基线一样新颖甚至更新颖，但模型仍然有时复制，在某些情况下，在训练集中重复超过1000字超过1,000字的通道。我们还表现了广泛的手动分析，表明GPT-2的新文本通常在形态学和语法中形成良好，但具有合理的语义问题（例如，是自相矛盾）。

快捷方式学习的问题在NLP中广为人知，并且近年来一直是重要的研究重点。数据中的意外相关性使模型能够轻松地求解旨在表现出高级语言理解和推理能力的任务。在本调查论文中，我们关注机器阅读理解的领域（MRC），这是展示高级语言理解的重要任务，这也遭受了一系列快捷方式。我们总结了用于测量和减轻快捷方式的可用技术，并以捷径研究进一步进展的建议结论。最重要的是，我们强调了MRC中缓解快捷方式的两个主要问题：缺乏公共挑战集，有效和可重复使用的评估的必要组成部分以及在其他领域中缺乏某些缓解技术。

For natural language understanding (NLU) technology to be maximally useful, it must be able to process language in a way that is not exclusive to a single task, genre, or dataset. In pursuit of this objective, we introduce the General Language Understanding Evaluation (GLUE) benchmark, a collection of tools for evaluating the performance of models across a diverse set of existing NLU tasks. By including tasks with limited training data, GLUE is designed to favor and encourage models that share general linguistic knowledge across tasks. GLUE also includes a hand-crafted diagnostic test suite that enables detailed linguistic analysis of models. We evaluate baselines based on current methods for transfer and representation learning and find that multi-task training on all tasks performs better than training a separate model per task. However, the low absolute performance of our best model indicates the need for improved general NLU systems.

大规模的预训练语言模型在广泛的自然语言理解（NLU）任务中取得了巨大的成功，甚至超过人类性能。然而，最近的研究表明，这些模型的稳健性可能受到精心制作的文本对抗例子的挑战。虽然已经提出了几个单独的数据集来评估模型稳健性，但仍缺少原则和全面的基准。在本文中，我们呈现对抗性胶水（AdvGlue），这是一个新的多任务基准，以定量和彻底探索和评估各种对抗攻击下现代大规模语言模型的脆弱性。特别是，我们系统地应用14种文本对抗的攻击方法来构建一个粘合的援助，这是由人类进一步验证的可靠注释。我们的调查结果总结如下。 （i）大多数现有的对抗性攻击算法容易发生无效或暧昧的对手示例，其中大约90％的含量改变原始语义含义或误导性的人的注册人。因此，我们执行仔细的过滤过程来策划高质量的基准。 （ii）我们测试的所有语言模型和强大的培训方法在AdvGlue上表现不佳，差价远远落后于良性准确性。我们希望我们的工作能够激励开发新的对抗攻击，这些攻击更加隐身，更加统一，以及针对复杂的对抗性攻击的新强大语言模型。 Advglue在https://adversarialglue.github.io提供。

数据增强是通过转换为机器学习的人工创建数据的人工创建，是一个跨机器学习学科的研究领域。尽管它对于增加模型的概括功能很有用，但它还可以解决许多其他挑战和问题，从克服有限的培训数据到正规化目标到限制用于保护隐私的数据的数量。基于对数据扩展的目标和应用的精确描述以及现有作品的分类法，该调查涉及用于文本分类的数据增强方法，并旨在为研究人员和从业者提供简洁而全面的概述。我们将100多种方法划分为12种不同的分组，并提供最先进的参考文献来阐述哪种方法可以通过将它们相互关联，从而阐述了哪种方法。最后，提供可能构成未来工作的基础的研究观点。

数据增强是自然语言处理（NLP）模型的鲁棒性评估的重要组成部分，以及增强他们培训的数据的多样性。在本文中，我们呈现NL-Cogmenter，这是一种新的参与式Python的自然语言增强框架，它支持创建两个转换（对数据的修改）和过滤器（根据特定功能的数据拆分）。我们描述了框架和初始的117个变换和23个过滤器，用于各种自然语言任务。我们通过使用其几个转换来分析流行自然语言模型的鲁棒性来证明NL-Upmenter的功效。基础架构，Datacards和稳健性分析结果在NL-Augmenter存储库上公开可用（\ url {https://github.com/gem-benchmark/nl-augmenter}）。

Multi-hop Machine reading comprehension is a challenging task with aim of answering a question based on disjoint pieces of information across the different passages. The evaluation metrics and datasets are a vital part of multi-hop MRC because it is not possible to train and evaluate models without them, also, the proposed challenges by datasets often are an important motivation for improving the existing models. Due to increasing attention to this field, it is necessary and worth reviewing them in detail. This study aims to present a comprehensive survey on recent advances in multi-hop MRC evaluation metrics and datasets. In this regard, first, the multi-hop MRC problem definition will be presented, then the evaluation metrics based on their multi-hop aspect will be investigated. Also, 15 multi-hop datasets have been reviewed in detail from 2017 to 2022, and a comprehensive analysis has been prepared at the end. Finally, open issues in this field have been discussed.

Grammatical Error Correction (GEC) is the task of automatically detecting and correcting errors in text. The task not only includes the correction of grammatical errors, such as missing prepositions and mismatched subject-verb agreement, but also orthographic and semantic errors, such as misspellings and word choice errors respectively. The field has seen significant progress in the last decade, motivated in part by a series of five shared tasks, which drove the development of rule-based methods, statistical classifiers, statistical machine translation, and finally neural machine translation systems which represent the current dominant state of the art. In this survey paper, we condense the field into a single article and first outline some of the linguistic challenges of the task, introduce the most popular datasets that are available to researchers (for both English and other languages), and summarise the various methods and techniques that have been developed with a particular focus on artificial error generation. We next describe the many different approaches to evaluation as well as concerns surrounding metric reliability, especially in relation to subjective human judgements, before concluding with an overview of recent progress and suggestions for future work and remaining challenges. We hope that this survey will serve as comprehensive resource for researchers who are new to the field or who want to be kept apprised of recent developments.

离散对手攻击是对保留输出标签的语言输入的象征性扰动，但导致预测误差。虽然这种攻击已经广泛探索了评估模型稳健性的目的，但他们的改善稳健性的效用仅限于离线增强。具体地，给定训练有素的模型，攻击用于产生扰动（对抗性）示例，并且模型重新培训一次。在这项工作中，我们解决了这个差距并利用了在线增强的离散攻击，在每个训练步骤中产生了对抗的例子，适应模型的变化性质。我们提出（i）基于最佳搜索的新的离散攻击，以及（ii）与现有工作不同的随机采样攻击不是基于昂贵的搜索过程。令人惊讶的是，我们发现随机抽样导致鲁棒性的令人印象深刻，优于普通使用的离线增强，同时导致训练时间〜10x的加速。此外，在线增强基于搜索的攻击证明了更高的培训成本，显着提高了三个数据集的鲁棒性。最后，我们表明我们的新攻击与先前的方法相比，我们的新攻击显着提高了鲁棒性。

近年来，低资源机器阅读理解（MRC）取得了重大进展，模型在各种语言数据集中获得了显着性能。但是，这些模型都没有为URDU语言定制。这项工作探讨了通过将机器翻译的队伍与来自剑桥O级书籍的Wikipedia文章和Urdu RC工作表组合的人生成的样本组合了机器翻译的小队，探讨了乌尔通题的半自动创建了数据集（UQuad1.0）。 UQuad1.0是一个大型URDU数据集，用于提取机器阅读理解任务，由49K问题答案成对组成，段落和回答格式。在UQuad1.0中，通过众包的原始SquAd1.0和大约4000对的机器翻译产生45000对QA。在本研究中，我们使用了两种类型的MRC型号：基于规则的基线和基于先进的变换器的模型。但是，我们发现后者优于其他人;因此，我们已经决定专注于基于变压器的架构。使用XLMroberta和多语言伯特，我们分别获得0.66和0.63的F1得分。

在过去的三年里，自动评分发动机已被用于评分大约五百万个测试者。由于Covid-19和相关的教育和测试自动化，这个数字进一步增加。尽管使用了这么广泛，但基于AI的测试文献非常缺乏。提出新模型的大多数论文仅依赖于基于二次加权的Kappa（QWK）与人类评估者的协议，以显示模型效能。然而，这有效地忽略了论文评分的高度多重特征性质。论文评分取决于相干性，语法，相关性，充足和，词汇等特征。迄今为止，没有研究检测自动化论文评分：AES系统在全面上的所有这些功能。通过这种动机，我们为AES系统提出了一种模型不良反对派评估计划和相关指标，以测试其自然语言的理解能力和整体鲁棒性。我们使用所提出的方案评估当前的最先进的AES模型，并在最近的五个模型上报告结果。这些型号范围从基于特征为本的最新深度学习算法的方法。我们发现AES模型是高度不夸张的。即使是重型修改（高达25％）与问题无关的内容也不会降低模型产生的分数。另一方面，平均不相关的内容增加了分数，从而表明应该重新考虑模型评估策略和尺寸。我们还要求200名人类评估者在看到人类可以检测到两者之间的差异以及是否同意自动分数分配的分数的同意，以获得原始和对抗的反应。