Dialect differences caused by regional, social, and economic barriers cause performance discrepancies for many groups of users of language technology. Fair, inclusive, and equitable language technology must critically be dialect invariant, meaning that performance remains constant over dialectal shifts. Current English systems often fall significantly short of this ideal since they are designed and tested on a single dialect: Standard American English. We introduce Multi-VALUE -- a suite of resources for evaluating and achieving English dialect invariance. We build a controllable rule-based translation system spanning 50 English dialects and a total of 189 unique linguistic features. Our translation maps Standard American English text to synthetic form of each dialect, which uses an upper-bound on the natural density of features in that dialect. First, we use this system to build stress tests for question answering, machine translation, and semantic parsing tasks. Stress tests reveal significant performance disparities for leading models on non-standard dialects. Second, we use this system as a data augmentation technique to improve the dialect robustness of existing systems. Finally, we partner with native speakers of Chicano and Indian English to release new gold-standard variants of the popular CoQA task.

英语自然语言理解（NLU）系统已经取得了出色的表现，甚至在胶水和超级胶水等基准上表现出色。但是，这些基准仅包含教科书标准美国英语（SAE）。在NLP社区中，其他方言在很大程度上被忽略了。这导致偏见且不平等的NLU系统，仅服务于说话者的子人群。为了了解当前模型的差异并促进了更多的语言功能性的NLU系统，我们介绍了白话语言理解评估（Value）基准，这是我们使用一套词汇和形态句法转换规则创建的具有挑战性的胶水变体。在此最初版本（v.1）中，我们为非裔美国人白话英语（AAVE）的11个特征构建规则，并招募流利的AAVE扬声器，以通过参与性设计方式通过语言可接受性判断来验证每个功能转换。实验表明，这些新的方言功能可以导致模型性能下降。要运行转换代码并下载合成和金标准的方言胶水标准，请参见https://github.com/salt-nlp/value

数据增强是自然语言处理（NLP）模型的鲁棒性评估的重要组成部分，以及增强他们培训的数据的多样性。在本文中，我们呈现NL-Cogmenter，这是一种新的参与式Python的自然语言增强框架，它支持创建两个转换（对数据的修改）和过滤器（根据特定功能的数据拆分）。我们描述了框架和初始的117个变换和23个过滤器，用于各种自然语言任务。我们通过使用其几个转换来分析流行自然语言模型的鲁棒性来证明NL-Upmenter的功效。基础架构，Datacards和稳健性分析结果在NL-Augmenter存储库上公开可用（\ url {https://github.com/gem-benchmark/nl-augmenter}）。

Grammatical Error Correction (GEC) is the task of automatically detecting and correcting errors in text. The task not only includes the correction of grammatical errors, such as missing prepositions and mismatched subject-verb agreement, but also orthographic and semantic errors, such as misspellings and word choice errors respectively. The field has seen significant progress in the last decade, motivated in part by a series of five shared tasks, which drove the development of rule-based methods, statistical classifiers, statistical machine translation, and finally neural machine translation systems which represent the current dominant state of the art. In this survey paper, we condense the field into a single article and first outline some of the linguistic challenges of the task, introduce the most popular datasets that are available to researchers (for both English and other languages), and summarise the various methods and techniques that have been developed with a particular focus on artificial error generation. We next describe the many different approaches to evaluation as well as concerns surrounding metric reliability, especially in relation to subjective human judgements, before concluding with an overview of recent progress and suggestions for future work and remaining challenges. We hope that this survey will serve as comprehensive resource for researchers who are new to the field or who want to be kept apprised of recent developments.

基于变压器的语言模型最近在许多自然语言任务中取得了显着的结果。但是，通常通过利用大量培训数据来实现排行榜的性能，并且很少通过将明确的语言知识编码为神经模型。这使许多人质疑语言学对现代自然语言处理的相关性。在本文中，我介绍了几个案例研究，以说明理论语言学和神经语言模型仍然相互关联。首先，语言模型通过提供一个客观的工具来测量语义距离，这对语言学家很有用，语义距离很难使用传统方法。另一方面，语言理论通过提供框架和数据源来探究我们的语言模型，以了解语言理解的特定方面，从而有助于语言建模研究。本论文贡献了三项研究，探讨了语言模型中语法 - 听觉界面的不同方面。在论文的第一部分中，我将语言模型应用于单词类灵活性的问题。我将Mbert作为语义距离测量的来源，我提供了有利于将单词类灵活性分析为方向过程的证据。在论文的第二部分中，我提出了一种方法来测量语言模型中间层的惊奇方法。我的实验表明，包含形态句法异常的句子触发了语言模型早期的惊喜，而不是语义和常识异常。最后，在论文的第三部分中，我适应了一些心理语言学研究，以表明语言模型包含了论证结构结构的知识。总而言之，我的论文在自然语言处理，语言理论和心理语言学之间建立了新的联系，以为语言模型的解释提供新的观点。

语言可以用作再现和执行有害刻板印象和偏差的手段，并被分析在许多研究中。在本文中，我们对自然语言处理中的性别偏见进行了304篇论文。我们分析了社会科学中性别及其类别的定义，并将其连接到NLP研究中性别偏见的正式定义。我们调查了在对性别偏见的研究中应用的Lexica和数据集，然后比较和对比方法来检测和减轻性别偏见。我们发现对性别偏见的研究遭受了四个核心限制。 1）大多数研究将性别视为忽视其流动性和连续性的二元变量。 2）大部分工作都在单机设置中进行英语或其他高资源语言进行。 3）尽管在NLP方法中对性别偏见进行了无数的论文，但我们发现大多数新开发的算法都没有测试他们的偏见模型，并无视他们的工作的伦理考虑。 4）最后，在这一研究线上发展的方法基本缺陷涵盖性别偏差的非常有限的定义，缺乏评估基线和管道。我们建议建议克服这些限制作为未来研究的指导。

本文介绍了正式和非正式波斯之间的语音，形态和句法区别，表明这两个变体具有根本差异，不能仅归因于发音差异。鉴于非正式波斯展出特殊的特征，任何在正式波斯语上培训的计算模型都不太可能转移到非正式的波斯，所以需要为这种品种创建专用的树木银行。因此，我们详细介绍了开源非正式波斯普通依赖性TreeBank的开发，这是一个在通用依赖性方案中注释的新的TreeBank。然后，我们通过在现有的正式树木银行上培训两个依赖性解析器并在域名数据上进行评估，调查非正式波斯的解析，即我们非正式树木银行的开发集。我们的结果表明，当我们穿过两个域时，解析器在跨越两个域时遇到了实质性的性能下降，因为它们面临更为不知名的令牌和结构，并且无法概括。此外，性能恶化的依赖关系最多代表了非正式变体的独特属性。这项研究的最终目标表明更广泛的影响是提供踩踏石头，以揭示语言的非正式变种的重要性，这被广泛地忽略了跨语言的自然语言处理工具。

数据饥饿的深度神经网络已经将自己作为许多NLP任务的标准建立为包括传统序列标记的标准。尽管他们在高资源语言上表现最先进的表现，但它们仍然落后于低资源场景的统计计数器。一个方法来反击攻击此问题是文本增强，即，从现有数据生成新的合成训练数据点。虽然NLP最近目睹了一种文本增强技术的负载，但该领域仍然缺乏对多种语言和序列标记任务的系统性能分析。为了填补这一差距，我们调查了三类文本增强方法，其在语法（例如，裁剪子句子），令牌（例如，随机字插入）和字符（例如，字符交换）级别上执行更改。我们系统地将它们与语音标记，依赖解析和语义角色标记的分组进行了比较，用于使用各种模型的各种语言系列，包括依赖于诸如MBERT的普赖金的多语言语境化语言模型的架构。增强最显着改善了解析，然后是语音标记和语义角色标记的依赖性解析。我们发现实验技术通常在形态上丰富的语言，而不是越南语等分析语言。我们的研究结果表明，增强技术可以进一步改善基于MBERT的强基线。我们将字符级方法标识为最常见的表演者，而同义词替换和语法增强仪提供不一致的改进。最后，我们讨论了最大依赖于任务，语言对和模型类型的结果。

In this work, we introduce IndicXTREME, a benchmark consisting of nine diverse tasks covering 18 languages from the Indic sub-continent belonging to four different families. Across languages and tasks, IndicXTREME contains a total of 103 evaluation sets, of which 51 are new contributions to the literature. To maintain high quality, we only use human annotators to curate or translate\footnote{for IndicXParaphrase, where an automatic translation system is used, a second human verification and correction step is done.} our datasets. To the best of our knowledge, this is the first effort toward creating a standard benchmark for Indic languages that aims to test the zero-shot capabilities of pretrained language models. We also release IndicCorp v2, an updated and much larger version of IndicCorp that contains 20.9 billion tokens in 24 languages. We pretrain IndicBERT v2 on IndicCorp v2 and evaluate it on IndicXTREME to show that it outperforms existing multilingual language models such as XLM-R and MuRIL.

尽管最近在机器学习用于自然语言处理的机器学习方面，但自然语言推论（NLI）问题仍然是挑战。为此目的，我们贡献了一个新的数据集，专注于事实现象;但是，我们的任务与其他NLI任务保持相同，即引起的征集，矛盾或中性（ECN）。 DataSet在波兰语中包含完全自然语言话语，收集2,432个动词补充对和309个独特的动词。 DataSet基于国家波兰语（NKJP）的国家语料库，是主要动词频率和其他语言特征的代表性样本（例如，内部否定的发生）。我们发现，基于变压器的基于判决的模型获得了相对良好的结果（$ \ \左右89 \％$ F1得分）。尽管使用语言特征实现了更好的结果（$ \大约91 \％$ F1得分），但这种模型需要更多人工劳动力（循环中的人类），因为专家语言学家手动制备特征。基于BERT的模型仅消耗输入句子表明，它们捕获了NLI / Factivity的大部分复杂性。现象中的复杂病例 - 例如具有权利（e）和非致命动词的案件 - 仍然是进一步研究的开放问题。

In the last year, new models and methods for pretraining and transfer learning have driven striking performance improvements across a range of language understanding tasks. The GLUE benchmark, introduced a little over one year ago, offers a single-number metric that summarizes progress on a diverse set of such tasks, but performance on the benchmark has recently surpassed the level of non-expert humans, suggesting limited headroom for further research. In this paper we present SuperGLUE, a new benchmark styled after GLUE with a new set of more difficult language understanding tasks, a software toolkit, and a public leaderboard. SuperGLUE is available at super.gluebenchmark.com.

Natural Language Generation (NLG) has improved exponentially in recent years thanks to the development of sequence-to-sequence deep learning technologies such as Transformer-based language models. This advancement has led to more fluent and coherent NLG, leading to improved development in downstream tasks such as abstractive summarization, dialogue generation and data-to-text generation. However, it is also apparent that deep learning based generation is prone to hallucinate unintended text, which degrades the system performance and fails to meet user expectations in many real-world scenarios. To address this issue, many studies have been presented in measuring and mitigating hallucinated texts, but these have never been reviewed in a comprehensive manner before. In this survey, we thus provide a broad overview of the research progress and challenges in the hallucination problem in NLG. The survey is organized into two parts: (1) a general overview of metrics, mitigation methods, and future directions; and (2) an overview of task-specific research progress on hallucinations in the following downstream tasks, namely abstractive summarization, dialogue generation, generative question answering, data-to-text generation, machine translation, and visual-language generation. This survey serves to facilitate collaborative efforts among researchers in tackling the challenge of hallucinated texts in NLG.

As machine translation (MT) metrics improve their correlation with human judgement every year, it is crucial to understand the limitations of such metrics at the segment level. Specifically, it is important to investigate metric behaviour when facing accuracy errors in MT because these can have dangerous consequences in certain contexts (e.g., legal, medical). We curate ACES, a translation accuracy challenge set, consisting of 68 phenomena ranging from simple perturbations at the word/character level to more complex errors based on discourse and real-world knowledge. We use ACES to evaluate a wide range of MT metrics including the submissions to the WMT 2022 metrics shared task and perform several analyses leading to general recommendations for metric developers. We recommend: a) combining metrics with different strengths, b) developing metrics that give more weight to the source and less to surface-level overlap with the reference and c) explicitly modelling additional language-specific information beyond what is available via multilingual embeddings.

自然语言推理（NLI）和语义文本相似性（STS）是广泛使用的基准任务，用于对预训练的语言模型进行组成评估。尽管对语言普遍性的兴趣越来越大，但大多数NLI/STS研究几乎完全集中在英语上。特别是，日语中没有可用的多语言NLI/STS数据集，它在类型上与英语不同，并且可以阐明语言模型当前有争议的行为，例如对单词顺序和案例粒子的敏感性。在此背景下，我们介绍了日本NLI/STS数据集Jsick，该数据集是从英语数据集病中手动翻译的。我们还提出了一个用于组成推断的应力测试数据集，该数据集是通过转换JSick中句子的句法结构来研究语言模型是否对单词顺序和案例粒子敏感的。我们在不同的预训练语言模型上进行基线实验，并比较应用于日语和其他语言时多语言模型的性能。应力测试实验的结果表明，当前的预训练的语言模型对单词顺序和案例标记不敏感。

在本文中，我们专注于研究中国问题匹配的鲁棒性评估。以前的大多数关于分析鲁棒性问题的工作专注于只有一种或几种类型的人工对抗例。相反，我们认为有必要制定关于自然文本模型语言能力的综合评估。为此目的，我们创建了一个中国数据集即duqm，其中包含具有语言扰动的自然问题，以评估问题匹配模型的鲁棒性。Duqm包含3个类别和13个子类别，具有32个语言扰动。广泛的实验表明，DUQM具有更好的区分不同模型的能力。重要的是，DuQM中语言现象评估的详细分类有助于我们轻松诊断不同模型的强度和弱点。此外，我们的实验结果表明，人工对抗实例的影响不适用于自然文本。

即使在高度发达的国家，多达15-30％的人口只能理解使用基本词汇编写的文本。他们对日常文本的理解是有限的，这阻止了他们在社会中发挥积极作用，并就医疗保健，法律代表或民主选择做出明智的决定。词汇简化是一项自然语言处理任务，旨在通过更简单地替换复杂的词汇和表达方式来使每个人都可以理解文本，同时保留原始含义。在过去的20年中，它引起了极大的关注，并且已经针对各种语言提出了全自动词汇简化系统。该领域进步的主要障碍是缺乏用于构建和评估词汇简化系统的高质量数据集。我们提出了一个新的基准数据集，用于英语，西班牙语和（巴西）葡萄牙语中的词汇简化，并提供有关数据选择和注释程序的详细信息。这是第一个可直接比较三种语言的词汇简化系统的数据集。为了展示数据集的可用性，我们将两种具有不同体系结构（神经与非神经）的最先进的词汇简化系统适应所有三种语言（英语，西班牙语和巴西葡萄牙语），并评估他们的表演在我们的新数据集中。为了进行更公平的比较，我们使用多种评估措施来捕获系统功效的各个方面，并讨论其优势和缺点。我们发现，最先进的神经词汇简化系统优于所有三种语言中最先进的非神经词汇简化系统。更重要的是，我们发现最先进的神经词汇简化系统对英语的表现要比西班牙和葡萄牙语要好得多。

Much recent progress in applications of machine learning models to NLP has been driven by benchmarks that evaluate models across a wide variety of tasks. However, these broad-coverage benchmarks have been mostly limited to English, and despite an increasing interest in multilingual models, a benchmark that enables the comprehensive evaluation of such methods on a diverse range of languages and tasks is still missing. To this end, we introduce the Cross-lingual TRansfer Evaluation of Multilingual Encoders (XTREME) benchmark, a multi-task benchmark for evaluating the cross-lingual generalization capabilities of multilingual representations across 40 languages and 9 tasks. We demonstrate that while models tested on English reach human performance on many tasks, there is still a sizable gap in the performance of cross-lingually transferred models, particularly on syntactic and sentence retrieval tasks. There is also a wide spread of results across languages. We release the benchmark 1 to encourage research on cross-lingual learning methods that transfer linguistic knowledge across a diverse and representative set of languages and tasks.

Question Answering (QA) is a growing area of research, often used to facilitate the extraction of information from within documents. State-of-the-art QA models are usually pre-trained on domain-general corpora like Wikipedia and thus tend to struggle on out-of-domain documents without fine-tuning. We demonstrate that synthetic domain-specific datasets can be generated easily using domain-general models, while still providing significant improvements to QA performance. We present two new tools for this task: A flexible pipeline for validating the synthetic QA data and training downstream models on it, and an online interface to facilitate human annotation of this generated data. Using this interface, crowdworkers labelled 1117 synthetic QA pairs, which we then used to fine-tune downstream models and improve domain-specific QA performance by 8.75 F1.

这项研究讨论了半监督学习的影响与验证的语言模型，以生成数据到文本。当还补充大规模语言模型时，尚不清楚半监督学习是否仍然有用。这项研究的目的是通过将仅补充语言模型的数据到文本系统与两个数据到文本系统进行比较，这些系统通过数据增强或伪标记的半固定学习方法而富含数据。结果表明，半监督学习会导致多样性指标的得分更高。在输出质量方面，使用伪标记方法扩展数据到文本系统的训练集确实提高了文本质量分数，但是数据增强方法在没有训练设置扩展的情况下得出了与系统相似的分数。这些结果表明，即使也存在语言模型，半监督的学习方法也可以增强产出质量和多样性。

We study politeness phenomena in nine typologically diverse languages. Politeness is an important facet of communication and is sometimes argued to be cultural-specific, yet existing computational linguistic study is limited to English. We create TyDiP, a dataset containing three-way politeness annotations for 500 examples in each language, totaling 4.5K examples. We evaluate how well multilingual models can identify politeness levels -- they show a fairly robust zero-shot transfer ability, yet fall short of estimated human accuracy significantly. We further study mapping the English politeness strategy lexicon into nine languages via automatic translation and lexicon induction, analyzing whether each strategy's impact stays consistent across languages. Lastly, we empirically study the complicated relationship between formality and politeness through transfer experiments. We hope our dataset will support various research questions and applications, from evaluating multilingual models to constructing polite multilingual agents.