We describe METEOR, an automatic metric for machine translation evaluation that is based on a generalized concept of unigram matching between the machineproduced translation and human-produced reference translations. Unigrams can be matched based on their surface forms, stemmed forms, and meanings; furthermore, METEOR can be easily extended to include more advanced matching strategies.Once all generalized unigram matches between the two strings have been found, METEOR computes a score for this matching using a combination of unigram-precision, unigram-recall, and a measure of fragmentation that is designed to directly capture how well-ordered the matched words in the machine translation are in relation to the reference. We evaluate METEOR by measuring the correlation between the metric scores and human judgments of translation quality. We compute the Pearson R correlation value between its scores and human quality assessments of the LDC TIDES 2003 Arabic-to-English and Chinese-to-English datasets.We perform segment-bysegment correlation, and show that METEOR gets an R correlation value of 0.347 on the Arabic data and 0.331 on the Chinese data. This is shown to be an improvement on using simply unigramprecision, unigram-recall and their harmonic F1 combination. We also perform experiments to show the relative contributions of the various mapping modules.

Human evaluations of machine translation are extensive but expensive. Human evaluations can take months to finish and involve human labor that can not be reused. We propose a method of automatic machine translation evaluation that is quick, inexpensive, and language-independent, that correlates highly with human evaluation, and that has little marginal cost per run. We present this method as an automated understudy to skilled human judges which substitutes for them when there is need for quick or frequent evaluations. 1

标准自动指标，例如BLEU对于文档级MT评估不可靠。他们既不能区分翻译质量的文档级改进与句子级别的改进，也不能确定引起上下文反应翻译的话语现象。本文介绍了一种新颖的自动公制金发，以扩大自动MT评估的范围，从句子到文档级别。金发女郎通过对与话语相关的跨度进行分类并计算基于相似性的F1分类跨度来考虑话语一致性。我们对新建的数据集BWB进行了广泛的比较。实验结果表明，金发女郎在文档级别具有更好的选择性和可解释性，并且对文档级别的细微差别更为敏感。在一项大规模的人类研究中，与以前的指标相比，金发碧眼的皮尔逊与人类判断的相关性也明显更高。

We propose BERTSCORE, an automatic evaluation metric for text generation. Analogously to common metrics, BERTSCORE computes a similarity score for each token in the candidate sentence with each token in the reference sentence. However, instead of exact matches, we compute token similarity using contextual embeddings. We evaluate using the outputs of 363 machine translation and image captioning systems. BERTSCORE correlates better with human judgments and provides stronger model selection performance than existing metrics. Finally, we use an adversarial paraphrase detection task to show that BERTSCORE is more robust to challenging examples when compared to existing metrics.

人类评估一直昂贵，而研究人员则努力信任自动指标。为了解决这个问题，我们建议通过采取预先接受训练的语言模型（PLM）和有限的人类标记分数来定制传统指标。我们首先重新介绍Hlepor度量因子，然后是我们开发的Python版本（移植），这实现了Hlepor度量中的加权参数的自动调整。然后我们介绍了使用Optuna超参数优化框架的定制Hlepor（Cushlepor），以便更好地协议为预先接受训练的语言模型（使用Labse），这是关于Cushlepor的确切MT语言对。我们还在英语 - 德语和汉英语言对基于MQM和PSQM框架的专业人体评估数据进行了优化的曲位波。实验研究表明，Cushlepor可以提升Hlepor对PLMS的更好的表演，如Labse，如Labse的更好的成本，以及更好的人类评估协议，包括MQM和PSQM得分，并且比Bleu（AT \ URL的数据提供更好的表演（HTTPS：// github.com/poethan/cushlepor}）。官方结果表明，我们的提交赢得了三种语言对，包括\ textbf {英语 - 德语}和\ textbf {中文 - 英文}通过cushlepor（lm）和\ textbf {英语 - 俄语}上\ textit {通过hlepor ted}域。

ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation. It includes measures to automatically determine the quality of a summary by comparing it to other (ideal) summaries created by humans. The measures count the number of overlapping units such as n-gram, word sequences, and word pairs between the computer-generated summary to be evaluated and the ideal summaries created by humans. This paper introduces four different ROUGE measures: ROUGE-N, ROUGE-L, ROUGE-W, and ROUGE-S included in the ROUGE summarization evaluation package and their evaluatio ns. Three of them have been used in the Document Understanding Conference (DUC) 2004, a large-scale summarization evaluation sponsored by NIST.

A lack of standard datasets and evaluation metrics has prevented the field of paraphrasing from making the kind of rapid progress enjoyed by the machine translation community over the last 15 years. We address both problems by presenting a novel data collection framework that produces highly parallel text data relatively inexpensively and on a large scale. The highly parallel nature of this data allows us to use simple n-gram comparisons to measure both the semantic adequacy and lexical dissimilarity of paraphrase candidates. In addition to being simple and efficient to compute, experiments show that these metrics correlate highly with human judgments.

This paper describes Meteor Universal, released for the 2014 ACL Workshop on Statistical Machine Translation. Meteor Universal brings language specific evaluation to previously unsupported target languages by (1) automatically extracting linguistic resources (paraphrase tables and function word lists) from the bitext used to train MT systems and (2) using a universal parameter set learned from pooling human judgments of translation quality from several language directions. Meteor Universal is shown to significantly outperform baseline BLEU on two new languages, Russian (WMT13) and Hindi (WMT14).

这项工作适用于最低贝叶斯风险（MBR）解码，以优化翻译质量的各种自动化指标。机器翻译中的自动指标最近取得了巨大的进步。特别是，在人类评级（例如BLEurt，或Comet）上微调，在与人类判断的相关性方面是优于表面度量的微调。我们的实验表明，神经翻译模型与神经基于基于神经参考度量，BLEURT的组合导致自动和人类评估的显着改善。通过与经典光束搜索输出不同的翻译获得该改进：这些翻译的可能性较低，并且较少受到Bleu等表面度量的青睐。

The field of machine translation faces an under-recognized problem because of inconsistency in the reporting of scores from its dominant metric. Although people refer to "the" BLEU score, BLEU is in fact a parameterized metric whose values can vary wildly with changes to these parameters. These parameters are often not reported or are hard to find, and consequently, BLEU scores between papers cannot be directly compared. I quantify this variation, finding differences as high as 1.8 between commonly used configurations. The main culprit is different tokenization and normalization schemes applied to the reference. Pointing to the success of the parsing community, I suggest machine translation researchers settle upon the BLEU scheme used by the annual Conference on Machine Translation (WMT), which does not allow for usersupplied reference processing, and provide a new tool, SACREBLEU, 1 to facilitate this.

我们为机器翻译（MT）评估发布了70个小鉴别的测试集，称为方差感知测试集（VAT），从WMT16覆盖了35个翻译方向到WMT20竞争。VAT由一种新颖的方差感知过滤方法自动创建，该方法会在没有任何人工的情况下过滤当前MT测试集的不分度测试实例。实验结果表明，VAT在主流语言对和测试集中与人为判断的相关性方面优于原始的WMT测试集。进一步分析增值税的性质揭示了竞争MT系统的具有挑战性的语言特征（例如，低频词和专有名词），为构建未来MT测试集提供指导。测试集和准备方差感知MT测试集的代码可在https://github.com/nlp2ct/variance-aware-mt-test-sets自由使用。

As machine translation (MT) metrics improve their correlation with human judgement every year, it is crucial to understand the limitations of such metrics at the segment level. Specifically, it is important to investigate metric behaviour when facing accuracy errors in MT because these can have dangerous consequences in certain contexts (e.g., legal, medical). We curate ACES, a translation accuracy challenge set, consisting of 68 phenomena ranging from simple perturbations at the word/character level to more complex errors based on discourse and real-world knowledge. We use ACES to evaluate a wide range of MT metrics including the submissions to the WMT 2022 metrics shared task and perform several analyses leading to general recommendations for metric developers. We recommend: a) combining metrics with different strengths, b) developing metrics that give more weight to the source and less to surface-level overlap with the reference and c) explicitly modelling additional language-specific information beyond what is available via multilingual embeddings.

The rapid growth of machine translation (MT) systems has necessitated comprehensive studies to meta-evaluate evaluation metrics being used, which enables a better selection of metrics that best reflect MT quality. Unfortunately, most of the research focuses on high-resource languages, mainly English, the observations for which may not always apply to other languages. Indian languages, having over a billion speakers, are linguistically different from English, and to date, there has not been a systematic study of evaluating MT systems from English into Indian languages. In this paper, we fill this gap by creating an MQM dataset consisting of 7000 fine-grained annotations, spanning 5 Indian languages and 7 MT systems, and use it to establish correlations between annotator scores and scores obtained using existing automatic metrics. Our results show that pre-trained metrics, such as COMET, have the highest correlations with annotator scores. Additionally, we find that the metrics do not adequately capture fluency-based errors in Indian languages, and there is a need to develop metrics focused on Indian languages. We hope that our dataset and analysis will help promote further research in this area.

Automatically describing an image with a sentence is a long-standing challenge in computer vision and natural language processing. Due to recent progress in object detection, attribute classification, action recognition, etc., there is renewed interest in this area. However, evaluating the quality of descriptions has proven to be challenging. We propose a novel paradigm for evaluating image descriptions that uses human consensus. This paradigm consists of three main parts: a new triplet-based method of collecting human annotations to measure consensus, a new automated metric (CIDEr) that captures consensus, and two new datasets: PASCAL-50S and ABSTRACT-50S that contain 50 sentences describing each image. Our simple metric captures human judgment of consensus better than existing metrics across sentences generated by various sources. We also evaluate five state-of-the-art image description approaches using this new protocol and provide a benchmark for future comparisons. A version of CIDEr named CIDEr-D is available as a part of MS COCO evaluation server to enable systematic evaluation and benchmarking.

估计机器翻译系统的质量是该领域的研究人员的持续挑战。许多以前使用往返翻译的尝试作为质量的衡量标准失败，并且对其是一种可行的质量估算方法有很大的分歧。在本文中，我们重新审视了往返翻译，提出了一个旨在解决这种方法发现的先前陷阱的系统。我们的方法利用近期语言表示的进步学习，以更准确地衡量原始和往返翻译句子之间的相似性。实验表明，虽然我们的方法没有达到现有技术的当前状态的性能，但它仍然可能是某些语言对的有效方法。

比较基准数据集的模型性能是人工智能测量和驱动进展的一个组成部分。模型在基准数据集上的性能通常基于单个或一小组性能指标进行评估。虽然这使得能够快速比较，但如果度量标准不充分覆盖所有性能特征，则可能导致模型性能不充分反映模型性能。它在多大程度上可能影响基准努力。为了解决这个问题，我们根据数据涵盖了3867个机器学习模型性能的基于数据，分析了当前的性能指标景观，从而用代码的开放存储库的“论文”。我们的研究结果表明，目前使用的大多数指标都有可能导致模型绩效反映不足的属性。虽然已经提出了解决有问题属性的替代度量，但目前很少使用它们。此外，我们描述了报告的指标中的歧义，这可能导致难以解释和比较模型表演。

Automatic machine translation (MT) metrics are widely used to distinguish the translation qualities of machine translation systems across relatively large test sets (system-level evaluation). However, it is unclear if automatic metrics are reliable at distinguishing good translations from bad translations at the sentence level (segment-level evaluation). In this paper, we investigate how useful MT metrics are at detecting the success of a machine translation component when placed in a larger platform with a downstream task. We evaluate the segment-level performance of the most widely used MT metrics (chrF, COMET, BERTScore, etc.) on three downstream cross-lingual tasks (dialogue state tracking, question answering, and semantic parsing). For each task, we only have access to a monolingual task-specific model. We calculate the correlation between the metric's ability to predict a good/bad translation with the success/failure on the final task for the Translate-Test setup. Our experiments demonstrate that all metrics exhibit negligible correlation with the extrinsic evaluation of the downstream outcomes. We also find that the scores provided by neural metrics are not interpretable mostly because of undefined ranges. Our analysis suggests that future MT metrics be designed to produce error labels rather than scores to facilitate extrinsic evaluation.

近年来，研究人员创建并引入了大量各种代码生成模型。由于对每个新模型版本的人类评估都是不可行的，因此社区采用了自动评估指标，例如BLEU来近似人类判断的结果。这些指标源自机器翻译域，目前尚不清楚它们是否适用于代码生成任务，以及他们与人类对此任务的评估有多一致。还有两个指标，即Codebleu和Ruby，它们是为了估计代码的相似性并考虑了代码属性的。但是，对于这些指标，几乎没有关于他们与人类评估一致的研究。尽管如此，公制得分的最小差异仍用于声称某些代码生成模型的优越性。在本文中，我们介绍了一项有关六个指标的适用性的研究-Bleu，Rouge-L，Meteor，Chrf，Codebleu，Ruby-用于评估代码生成模型。我们对两个不同的代码生成数据集进行了一项研究，并使用人类注释来评估这些数据集上运行的所有模型的质量。结果表明，对于Python单线的Conala数据集，如果模型得分的差异小于5分，则没有一个指标可以正确模拟人类判断，而$ 95 \％$确定性，则使用$> 95 \％$确定性。对于由特定结构类别组成的炉石传说数据集，至少2分的模型得分差异足以声称一种模型比另一个模型的优越性。使用我们的发现，我们得出了有关使用指标来估计代码生成任务的模型性能的几项建议。

本报告介绍了第七次机器翻译会议（WMT22）的通用机器翻译任务的自动评估。它总共评估了21个翻译方向的185个系统，包括高资源至低资源语言对以及与遥远语言密切相关的系统。这种大规模的自动评估突出了最新机器翻译系统的一些当前限制。它还显示了自动指标，即CHRF，BLEU和COMET，如何在解释性和准确性方面进行补充以减轻自己的限制。

评估指标是文本生成系统的关键成分。近年来，已经提出了几十年前的文本生成质量的人类评估，提出了几个基于伯特的评估指标（包括Bertscore，Moverscore，BLEurt等），这些评估与文本生成质量的人类评估比Bleu或Rouge进行了更好。但是，很少是已知这些度量基于黑盒语言模型表示的指标实际捕获（通常假设它们模型语义相似性）。在这项工作中，我们使用基于简单的回归的全局解释技术来沿着语言因素解开度量标准分数，包括语义，语法，形态和词汇重叠。我们表明，不同的指标捕获了一定程度的各个方面，但它们对词汇重叠大大敏感，就像Bleu和Rouge一样。这暴露了这些新颖性拟议的指标的限制，我们还在对抗对抗测试场景中突出显示。