与人类的比较是基准的基本要求是它是一种可靠的模型能力测量。然而，模型比较方法可能具有基础缺陷 - 单独度量的算术平均值用于不同复杂性的所有任务，测试和训练集的不同大小。在本文中，我们根据其报告的结果，检查流行的NLP基准测试的整体评分方法，并通过几何和谐波（适于平均率）重新排列模型。我们分析了几个流行的基准，包括胶水，超级格，XGLUE和Xtreme。分析表明，例如，SuperGlue上的人类水平仍未到达，目前模型还有改进的余地。

比较基准数据集的模型性能是人工智能测量和驱动进展的一个组成部分。模型在基准数据集上的性能通常基于单个或一小组性能指标进行评估。虽然这使得能够快速比较，但如果度量标准不充分覆盖所有性能特征，则可能导致模型性能不充分反映模型性能。它在多大程度上可能影响基准努力。为了解决这个问题，我们根据数据涵盖了3867个机器学习模型性能的基于数据，分析了当前的性能指标景观，从而用代码的开放存储库的“论文”。我们的研究结果表明，目前使用的大多数指标都有可能导致模型绩效反映不足的属性。虽然已经提出了解决有问题属性的替代度量，但目前很少使用它们。此外，我们描述了报告的指标中的歧义，这可能导致难以解释和比较模型表演。

Due to their crucial role in all NLP, several benchmarks have been proposed to evaluate pretrained language models. In spite of these efforts, no public benchmark of diverse nature currently exists for evaluation of Arabic. This makes it challenging to measure progress for both Arabic and multilingual language models. This challenge is compounded by the fact that any benchmark targeting Arabic needs to take into account the fact that Arabic is not a single language but rather a collection of languages and varieties. In this work, we introduce ORCA, a publicly available benchmark for Arabic language understanding evaluation. ORCA is carefully constructed to cover diverse Arabic varieties and a wide range of challenging Arabic understanding tasks exploiting 60 different datasets across seven NLU task clusters. To measure current progress in Arabic NLU, we use ORCA to offer a comprehensive comparison between 18 multilingual and Arabic language models. We also provide a public leaderboard with a unified single-number evaluation metric (ORCA score) to facilitate future research.

Pre-trained language models (PLMs) are known to improve the generalization performance of natural language understanding models by leveraging large amounts of data during the pre-training phase. However, the out-of-distribution (OOD) generalization problem remains a challenge in many NLP tasks, limiting the real-world deployment of these methods. This paper presents the first attempt at creating a unified benchmark named GLUE-X for evaluating OOD robustness in NLP models, highlighting the importance of OOD robustness and providing insights on how to measure the robustness of a model and how to improve it. The benchmark includes 13 publicly available datasets for OOD testing, and evaluations are conducted on 8 classic NLP tasks over 19 popularly used PLMs. Our findings confirm the need for improved OOD accuracy in NLP tasks, as significant performance degradation was observed in all settings compared to in-distribution (ID) accuracy.

Laws and their interpretations, legal arguments and agreements\ are typically expressed in writing, leading to the production of vast corpora of legal text. Their analysis, which is at the center of legal practice, becomes increasingly elaborate as these collections grow in size. Natural language understanding (NLU) technologies can be a valuable tool to support legal practitioners in these endeavors. Their usefulness, however, largely depends on whether current state-of-the-art models can generalize across various tasks in the legal domain. To answer this currently open question, we introduce the Legal General Language Understanding Evaluation (LexGLUE) benchmark, a collection of datasets for evaluating model performance across a diverse set of legal NLU tasks in a standardized way. We also provide an evaluation and analysis of several generic and legal-oriented models demonstrating that the latter consistently offer performance improvements across multiple tasks.

即使在高度发达的国家，多达15-30％的人口只能理解使用基本词汇编写的文本。他们对日常文本的理解是有限的，这阻止了他们在社会中发挥积极作用，并就医疗保健，法律代表或民主选择做出明智的决定。词汇简化是一项自然语言处理任务，旨在通过更简单地替换复杂的词汇和表达方式来使每个人都可以理解文本，同时保留原始含义。在过去的20年中，它引起了极大的关注，并且已经针对各种语言提出了全自动词汇简化系统。该领域进步的主要障碍是缺乏用于构建和评估词汇简化系统的高质量数据集。我们提出了一个新的基准数据集，用于英语，西班牙语和（巴西）葡萄牙语中的词汇简化，并提供有关数据选择和注释程序的详细信息。这是第一个可直接比较三种语言的词汇简化系统的数据集。为了展示数据集的可用性，我们将两种具有不同体系结构（神经与非神经）的最先进的词汇简化系统适应所有三种语言（英语，西班牙语和巴西葡萄牙语），并评估他们的表演在我们的新数据集中。为了进行更公平的比较，我们使用多种评估措施来捕获系统功效的各个方面，并讨论其优势和缺点。我们发现，最先进的神经词汇简化系统优于所有三种语言中最先进的非神经词汇简化系统。更重要的是，我们发现最先进的神经词汇简化系统对英语的表现要比西班牙和葡萄牙语要好得多。

Abstractive summarization has enjoyed renewed interest in recent years, thanks to pre-trained language models and the availability of large-scale datasets. Despite promising results, current models still suffer from generating factually inconsistent summaries, reducing their utility for real-world application. Several recent efforts attempt to address this by devising models that automatically detect factual inconsistencies in machine generated summaries. However, they focus exclusively on English, a language with abundant resources. In this work, we leverage factual consistency evaluation models to improve multilingual summarization. We explore two intuitive approaches to mitigate hallucinations based on the signal provided by a multilingual NLI model, namely data filtering and controlled generation. Experimental results in the 45 languages from the XLSum dataset show gains over strong baselines in both automatic and human evaluation.

我们认为当前的红外标准，用于优化用户体验，测量太窄的IR空间的一部分。如果IR系统较弱，这些指标缺乏或完全过滤出需要改进的更深层次的文件。如果IR系统相对强，则这些指标欠更深的相关文档，这些文档可以在用户可消化的层次结构或文本摘要中呈现出甚至更强大的IR系统，这些文件甚至可以呈现来自数十或数百个相关文档的内容。我们从过去28年重新分析了超过70个TREC曲目，显示大约一半的欠压排名的文件，几乎所有的缺乏尾部文件。我们展示在2020年的深度学习轨道中，神经系统在排名第一的文件中实际上是近乎最佳的，而在尾部文件上只有BM25的适度增益相比。我们的分析基于简单的新系统导向度量，“雾化搜索长度”，它能够在任何深度准确且均匀地测量所有相关文档。

人类评估一直昂贵，而研究人员则努力信任自动指标。为了解决这个问题，我们建议通过采取预先接受训练的语言模型（PLM）和有限的人类标记分数来定制传统指标。我们首先重新介绍Hlepor度量因子，然后是我们开发的Python版本（移植），这实现了Hlepor度量中的加权参数的自动调整。然后我们介绍了使用Optuna超参数优化框架的定制Hlepor（Cushlepor），以便更好地协议为预先接受训练的语言模型（使用Labse），这是关于Cushlepor的确切MT语言对。我们还在英语 - 德语和汉英语言对基于MQM和PSQM框架的专业人体评估数据进行了优化的曲位波。实验研究表明，Cushlepor可以提升Hlepor对PLMS的更好的表演，如Labse，如Labse的更好的成本，以及更好的人类评估协议，包括MQM和PSQM得分，并且比Bleu（AT \ URL的数据提供更好的表演（HTTPS：// github.com/poethan/cushlepor}）。官方结果表明，我们的提交赢得了三种语言对，包括\ textbf {英语 - 德语}和\ textbf {中文 - 英文}通过cushlepor（lm）和\ textbf {英语 - 俄语}上\ textit {通过hlepor ted}域。

Human evaluation is the foundation upon which the evaluation of both summarization systems and automatic metrics rests. However, existing human evaluation protocols and benchmarks for summarization either exhibit low inter-annotator agreement or lack the scale needed to draw statistically significant conclusions, and an in-depth analysis of human evaluation is lacking. In this work, we address the shortcomings of existing summarization evaluation along the following axes: 1) We propose a modified summarization salience protocol, Atomic Content Units (ACUs), which relies on fine-grained semantic units and allows for high inter-annotator agreement. 2) We curate the Robust Summarization Evaluation (RoSE) benchmark, a large human evaluation dataset consisting of over 22k summary-level annotations over state-of-the-art systems on three datasets. 3) We compare our ACU protocol with three other human evaluation protocols, underscoring potential confounding factors in evaluation setups. 4) We evaluate existing automatic metrics using the collected human annotations across evaluation protocols and demonstrate how our benchmark leads to more statistically stable and significant results. Furthermore, our findings have important implications for evaluating large language models (LLMs), as we show that LLMs adjusted by human feedback (e.g., GPT-3.5) may overfit unconstrained human evaluation, which is affected by the annotators' prior, input-agnostic preferences, calling for more robust, targeted evaluation methods.

最近提出的基于BERT的评估指标在标准评估基准方面表现良好，但容易受到对抗性攻击的影响，例如与事实错误有关。我们认为这（部分原因）是因为它们是语义相似性的模型。相反，我们根据自然语言推断（NLI）制定评估指标，我们认为这是更合适的建模。我们设计了一个基于偏好的对抗攻击框架，并表明我们的基于NLI的指标比最近基于BERT的指标更强大。在标准基准上，我们的基于NLI的指标的表现优于现有的摘要指标，但在SOTA MT指标下执行。但是，当我们将现有指标与NLI指标相结合时，我们可以获得更高的对抗性鲁棒性（ +20％至 +30％）和较高质量的指标，如标准基准测量（ +5％至 +25％）。

In the last year, new models and methods for pretraining and transfer learning have driven striking performance improvements across a range of language understanding tasks. The GLUE benchmark, introduced a little over one year ago, offers a single-number metric that summarizes progress on a diverse set of such tasks, but performance on the benchmark has recently surpassed the level of non-expert humans, suggesting limited headroom for further research. In this paper we present SuperGLUE, a new benchmark styled after GLUE with a new set of more difficult language understanding tasks, a software toolkit, and a public leaderboard. SuperGLUE is available at super.gluebenchmark.com.

NLP基准在很大程度上主要集中在短篇文本上，例如句子和段落，即使长文本在野外占相当数量的自然语言。我们介绍卷轴，这是一套需要在长文本上推理的任务套件。我们检查现有的长文本数据集，文本自然是长期的，同时优先考虑涉及在输入上扫描信息的任务。滚动包含概述，问题应答和自然语言推理任务，包括多个域，包括文学，科学，业务和娱乐。初始基线（包括啰覆编码器），表明滚动有充足的改进空间。我们以统一的文本到文本格式提供所有数据集，并托管Live Refordboard，以促进模型架构和预用方法的研究。

Powerful generative models have led to recent progress in question generation (QG). However, it is difficult to measure advances in QG research since there are no standardized resources that allow a uniform comparison among approaches. In this paper, we introduce QG-Bench, a multilingual and multidomain benchmark for QG that unifies existing question answering datasets by converting them to a standard QG setting. It includes general-purpose datasets such as SQuAD for English, datasets from ten domains and two styles, as well as datasets in eight different languages. Using QG-Bench as a reference, we perform an extensive analysis of the capabilities of language models for the task. First, we propose robust QG baselines based on fine-tuning generative language models. Then, we complement automatic evaluation based on standard metrics with an extensive manual evaluation, which in turn sheds light on the difficulty of evaluating QG models. Finally, we analyse both the domain adaptability of these models as well as the effectiveness of multilingual models in languages other than English. QG-Bench is released along with the fine-tuned models presented in the paper https://github.com/asahi417/lm-question-generation, which are also available as a demo https://autoqg.net/.

We describe METEOR, an automatic metric for machine translation evaluation that is based on a generalized concept of unigram matching between the machineproduced translation and human-produced reference translations. Unigrams can be matched based on their surface forms, stemmed forms, and meanings; furthermore, METEOR can be easily extended to include more advanced matching strategies.Once all generalized unigram matches between the two strings have been found, METEOR computes a score for this matching using a combination of unigram-precision, unigram-recall, and a measure of fragmentation that is designed to directly capture how well-ordered the matched words in the machine translation are in relation to the reference. We evaluate METEOR by measuring the correlation between the metric scores and human judgments of translation quality. We compute the Pearson R correlation value between its scores and human quality assessments of the LDC TIDES 2003 Arabic-to-English and Chinese-to-English datasets.We perform segment-bysegment correlation, and show that METEOR gets an R correlation value of 0.347 on the Arabic data and 0.331 on the Chinese data. This is shown to be an improvement on using simply unigramprecision, unigram-recall and their harmonic F1 combination. We also perform experiments to show the relative contributions of the various mapping modules.

AI中的不同子场倾向于储存一小部分有影响力的基准。这些基准作为一系列涂抹的常见问题的支架运作，这些常见问题经常被录制为朝向灵活和更广泛的AI系统的道路上的基础里程碑。这些基准最先进的性能被广泛理解为表明对这些长期目标的进展。在这个位置纸中，我们探讨了这种基准的限制，以便在其框架中揭示构建有效性问题，作为功能“一般”的进展措施，他们被设置为。

致辞推理是自然语言处理中的关键问题之一，但标记数据的相对稀缺缺少英语以外的语言的进度。预先磨削的交叉模型是一种强大的语言不可知论者的来源，但它们的固有推理能力仍在积极研究。在这项工作中，我们设计了一种简单的方法来推理，将线性分类器列举为具有多针关注的重量。为了评估这种方法，我们通过在标准化管道内的先前工作中处理多种数据集来创建多语言WinoGrad模式语料库，并在样品外性能方面测量交叉语言泛化能力。该方法在近期监督和无人监督推理的最近监督和无监督的方法中表现得很竞争，即使在以零拍摄方式应用于其他语言。此外，我们证明大多数性能由所有研究的语言的相同小的注意头给出，这提供了在多语言编码器中的普遍推理能力的证据。

In this work, we introduce IndicXTREME, a benchmark consisting of nine diverse tasks covering 18 languages from the Indic sub-continent belonging to four different families. Across languages and tasks, IndicXTREME contains a total of 103 evaluation sets, of which 51 are new contributions to the literature. To maintain high quality, we only use human annotators to curate or translate\footnote{for IndicXParaphrase, where an automatic translation system is used, a second human verification and correction step is done.} our datasets. To the best of our knowledge, this is the first effort toward creating a standard benchmark for Indic languages that aims to test the zero-shot capabilities of pretrained language models. We also release IndicCorp v2, an updated and much larger version of IndicCorp that contains 20.9 billion tokens in 24 languages. We pretrain IndicBERT v2 on IndicCorp v2 and evaluate it on IndicXTREME to show that it outperforms existing multilingual language models such as XLM-R and MuRIL.

Grammatical Error Correction (GEC) is the task of automatically detecting and correcting errors in text. The task not only includes the correction of grammatical errors, such as missing prepositions and mismatched subject-verb agreement, but also orthographic and semantic errors, such as misspellings and word choice errors respectively. The field has seen significant progress in the last decade, motivated in part by a series of five shared tasks, which drove the development of rule-based methods, statistical classifiers, statistical machine translation, and finally neural machine translation systems which represent the current dominant state of the art. In this survey paper, we condense the field into a single article and first outline some of the linguistic challenges of the task, introduce the most popular datasets that are available to researchers (for both English and other languages), and summarise the various methods and techniques that have been developed with a particular focus on artificial error generation. We next describe the many different approaches to evaluation as well as concerns surrounding metric reliability, especially in relation to subjective human judgements, before concluding with an overview of recent progress and suggestions for future work and remaining challenges. We hope that this survey will serve as comprehensive resource for researchers who are new to the field or who want to be kept apprised of recent developments.

In this work, we present an evaluation of smaller BLOOM model variants (350m/560m and 1b3/1b7) on various natural language processing tasks. This includes GLUE - language understanding, prompt-based zero-shot and few-shot text classification and extraction, question answering, prompt-based text generation, and multi-lingual text classification to understand model strengths/weaknesses and behavior. Empirical results show that BLOOM variants under-perform on all GLUE tasks (except WNLI), question-answering, and text generation. The variants bloom for WNLI, with an accuracy of 56.3%, and for prompt-based few-shot text extraction on MIT Movies and ATIS datasets. The BLOOM variants on average have 7% greater accuracy over GPT-2 and GPT-Neo models on Director and Airline Name extraction from MIT Movies and ATIS datasets, respectively.