While the problem of hallucinations in neural machine translation has long been recognized, so far the progress on its alleviation is very little. Indeed, recently it turned out that without artificially encouraging models to hallucinate, previously existing methods fall short and even the standard sequence log-probability is more informative. It means that characteristics internal to the model can give much more information than we expect, and before using external models and measures, we first need to ask: how far can we go if we use nothing but the translation model itself ? We propose to use a method that evaluates the percentage of the source contribution to a generated translation. Intuitively, hallucinations are translations "detached" from the source, hence they can be identified by low source contribution. This method improves detection accuracy for the most severe hallucinations by a factor of 2 and is able to alleviate hallucinations at test time on par with the previous best approach that relies on external models. Next, if we move away from internal model characteristics and allow external tools, we show that using sentence similarity from cross-lingual embeddings further improves these results.

尽管神经机器翻译（NMT）中幻觉的问题受到了一些关注，但对这种高度病理现象的研究缺乏坚实的基础。以前的工作在几种方面受到限制：它通常诉诸于放大问题的人工环境，它无视一些（常见的）幻觉类型，并且不能验证检测启发式方法的充分性。在本文中，我们为研究NMT幻觉的研究设定了基础。首先，我们在自然环境中工作，即没有人造噪声的内域数据，既不在训练中也没有推理。接下来，我们注释一个超过3.4K句子的数据集，指示不同类型的关键错误和幻觉。然后，我们转向以前使用的检测方法和两种重新访问方法，并建议使用基于玻璃盒的不确定性检测器。总体而言，我们表明，对于预防性设置，（i）先前使用的方法在很大程度上不足，（ii）序列对数概要性效果最好，并且与基于参考的方法相同。最后，我们提出了脱足素剂，这是一种减轻测试时间的简单方法，可大大降低幻觉速度。为了简化未来的研究，我们发布了用于WMT18德语英语数据的注释数据集以及模型，培训数据和代码。

Neural machine translation (NMT) has become the de-facto standard in real-world machine translation applications. However, NMT models can unpredictably produce severely pathological translations, known as hallucinations, that seriously undermine user trust. It becomes thus crucial to implement effective preventive strategies to guarantee their proper functioning. In this paper, we address the problem of hallucination detection in NMT by following a simple intuition: as hallucinations are detached from the source content, they exhibit encoder-decoder attention patterns that are statistically different from those of good quality translations. We frame this problem with an optimal transport formulation and propose a fully unsupervised, plug-in detector that can be used with any attention-based NMT model. Experimental results show that our detector not only outperforms all previous model-based detectors, but is also competitive with detectors that employ large models trained on millions of samples.

这项工作适用于最低贝叶斯风险（MBR）解码，以优化翻译质量的各种自动化指标。机器翻译中的自动指标最近取得了巨大的进步。特别是，在人类评级（例如BLEurt，或Comet）上微调，在与人类判断的相关性方面是优于表面度量的微调。我们的实验表明，神经翻译模型与神经基于基于神经参考度量，BLEURT的组合导致自动和人类评估的显着改善。通过与经典光束搜索输出不同的翻译获得该改进：这些翻译的可能性较低，并且较少受到Bleu等表面度量的青睐。

As machine translation (MT) metrics improve their correlation with human judgement every year, it is crucial to understand the limitations of such metrics at the segment level. Specifically, it is important to investigate metric behaviour when facing accuracy errors in MT because these can have dangerous consequences in certain contexts (e.g., legal, medical). We curate ACES, a translation accuracy challenge set, consisting of 68 phenomena ranging from simple perturbations at the word/character level to more complex errors based on discourse and real-world knowledge. We use ACES to evaluate a wide range of MT metrics including the submissions to the WMT 2022 metrics shared task and perform several analyses leading to general recommendations for metric developers. We recommend: a) combining metrics with different strengths, b) developing metrics that give more weight to the source and less to surface-level overlap with the reference and c) explicitly modelling additional language-specific information beyond what is available via multilingual embeddings.

We propose BERTSCORE, an automatic evaluation metric for text generation. Analogously to common metrics, BERTSCORE computes a similarity score for each token in the candidate sentence with each token in the reference sentence. However, instead of exact matches, we compute token similarity using contextual embeddings. We evaluate using the outputs of 363 machine translation and image captioning systems. BERTSCORE correlates better with human judgments and provides stronger model selection performance than existing metrics. Finally, we use an adversarial paraphrase detection task to show that BERTSCORE is more robust to challenging examples when compared to existing metrics.

神经指标与机器翻译系统评估中的人类判断达到了令人印象深刻的相关性，但是在我们可以安全地针对此类指标进行优化之前，我们应该意识到（并且理想地消除）偏向获得高分的不良翻译的偏见。我们的实验表明，基于样本的最小贝叶斯风险解码可用于探索和量化此类弱点。在将此策略应用于彗星进行ende和de-en时，我们发现彗星模型不足以差异和命名实体差异。我们进一步表明，通过简单地培训其他合成数据并发布我们的代码和数据以促进进一步的实验，这些偏见很难完全消除。

Automatic machine translation (MT) metrics are widely used to distinguish the translation qualities of machine translation systems across relatively large test sets (system-level evaluation). However, it is unclear if automatic metrics are reliable at distinguishing good translations from bad translations at the sentence level (segment-level evaluation). In this paper, we investigate how useful MT metrics are at detecting the success of a machine translation component when placed in a larger platform with a downstream task. We evaluate the segment-level performance of the most widely used MT metrics (chrF, COMET, BERTScore, etc.) on three downstream cross-lingual tasks (dialogue state tracking, question answering, and semantic parsing). For each task, we only have access to a monolingual task-specific model. We calculate the correlation between the metric's ability to predict a good/bad translation with the success/failure on the final task for the Translate-Test setup. Our experiments demonstrate that all metrics exhibit negligible correlation with the extrinsic evaluation of the downstream outcomes. We also find that the scores provided by neural metrics are not interpretable mostly because of undefined ranges. Our analysis suggests that future MT metrics be designed to produce error labels rather than scores to facilitate extrinsic evaluation.

我们假设现有的句子级机器翻译（MT）指标在人类参考包含歧义时会效率降低。为了验证这一假设，我们提出了一种非常简单的方法，用于扩展预审计的指标以在文档级别合并上下文。我们将我们的方法应用于三个流行的指标，即Bertscore，Prism和Comet，以及无参考的公制Comet-QE。我们使用提供的MQM注释评估WMT 2021指标共享任务的扩展指标。我们的结果表明，扩展指标的表现在约85％的测试条件下优于其句子级别的级别，而在排除低质量人类参考的结果时。此外，我们表明我们的文档级扩展大大提高了其对话语现象任务的准确性，从而优于专用基线高达6.1％。我们的实验结果支持我们的初始假设，并表明对指标的简单扩展使他们能够利用上下文来解决参考中的歧义。

The word alignment task, despite its prominence in the era of statistical machine translation (SMT), is niche and under-explored today. In this two-part tutorial, we argue for the continued relevance for word alignment. The first part provides a historical background to word alignment as a core component of the traditional SMT pipeline. We zero-in on GIZA++, an unsupervised, statistical word aligner with surprising longevity. Jumping forward to the era of neural machine translation (NMT), we show how insights from word alignment inspired the attention mechanism fundamental to present-day NMT. The second part shifts to a survey approach. We cover neural word aligners, showing the slow but steady progress towards surpassing GIZA++ performance. Finally, we cover the present-day applications of word alignment, from cross-lingual annotation projection, to improving translation.

Translating training data into many languages has emerged as a practical solution for improving cross-lingual transfer. For tasks that involve span-level annotations, such as information extraction or question answering, an additional label projection step is required to map annotated spans onto the translated texts. Recently, a few efforts have utilized a simple mark-then-translate method to jointly perform translation and projection by inserting special markers around the labeled spans in the original sentence. However, as far as we are aware, no empirical analysis has been conducted on how this approach compares to traditional annotation projection based on word alignment. In this paper, we present an extensive empirical study across 42 languages and three tasks (QA, NER, and Event Extraction) to evaluate the effectiveness and limitations of both methods, filling an important gap in the literature. Experimental results show that our optimized version of mark-then-translate, which we call EasyProject, is easily applied to many languages and works surprisingly well, outperforming the more complex word alignment-based methods. We analyze several key factors that affect end-task performance, and show EasyProject works well because it can accurately preserve label span boundaries after translation. We will publicly release all our code and data.

Natural Language Generation (NLG) has improved exponentially in recent years thanks to the development of sequence-to-sequence deep learning technologies such as Transformer-based language models. This advancement has led to more fluent and coherent NLG, leading to improved development in downstream tasks such as abstractive summarization, dialogue generation and data-to-text generation. However, it is also apparent that deep learning based generation is prone to hallucinate unintended text, which degrades the system performance and fails to meet user expectations in many real-world scenarios. To address this issue, many studies have been presented in measuring and mitigating hallucinated texts, but these have never been reviewed in a comprehensive manner before. In this survey, we thus provide a broad overview of the research progress and challenges in the hallucination problem in NLG. The survey is organized into two parts: (1) a general overview of metrics, mitigation methods, and future directions; and (2) an overview of task-specific research progress on hallucinations in the following downstream tasks, namely abstractive summarization, dialogue generation, generative question answering, data-to-text generation, machine translation, and visual-language generation. This survey serves to facilitate collaborative efforts among researchers in tackling the challenge of hallucinated texts in NLG.

In the absence of readily available labeled data for a given task and language, annotation projection has been proposed as one of the possible strategies to automatically generate annotated data which may then be used to train supervised systems. Annotation projection has often been formulated as the task of projecting, on parallel corpora, some labels from a source into a target language. In this paper we present T-Projection, a new approach for annotation projection that leverages large pretrained text2text language models and state-of-the-art machine translation technology. T-Projection decomposes the label projection task into two subtasks: (i) The candidate generation step, in which a set of projection candidates using a multilingual T5 model is generated and, (ii) the candidate selection step, in which the candidates are ranked based on translation probabilities. We evaluate our method in three downstream tasks and five different languages. Our results show that T-projection improves the average F1 score of previous methods by more than 8 points.

The rapid growth of machine translation (MT) systems has necessitated comprehensive studies to meta-evaluate evaluation metrics being used, which enables a better selection of metrics that best reflect MT quality. Unfortunately, most of the research focuses on high-resource languages, mainly English, the observations for which may not always apply to other languages. Indian languages, having over a billion speakers, are linguistically different from English, and to date, there has not been a systematic study of evaluating MT systems from English into Indian languages. In this paper, we fill this gap by creating an MQM dataset consisting of 7000 fine-grained annotations, spanning 5 Indian languages and 7 MT systems, and use it to establish correlations between annotator scores and scores obtained using existing automatic metrics. Our results show that pre-trained metrics, such as COMET, have the highest correlations with annotator scores. Additionally, we find that the metrics do not adequately capture fluency-based errors in Indian languages, and there is a need to develop metrics focused on Indian languages. We hope that our dataset and analysis will help promote further research in this area.

State-of-the-art natural language processing systems rely on supervision in the form of annotated data to learn competent models. These models are generally trained on data in a single language (usually English), and cannot be directly used beyond that language. Since collecting data in every language is not realistic, there has been a growing interest in crosslingual language understanding (XLU) and low-resource cross-language transfer. In this work, we construct an evaluation set for XLU by extending the development and test sets of the Multi-Genre Natural Language Inference Corpus (MultiNLI) to 15 languages, including low-resource languages such as Swahili and Urdu. We hope that our dataset, dubbed XNLI, will catalyze research in cross-lingual sentence understanding by providing an informative standard evaluation task. In addition, we provide several baselines for multilingual sentence understanding, including two based on machine translation systems, and two that use parallel data to train aligned multilingual bag-of-words and LSTM encoders. We find that XNLI represents a practical and challenging evaluation suite, and that directly translating the test data yields the best performance among available baselines.

翻译质量估计（QE）是预测机器翻译（MT）输出质量的任务，而无需任何参考。作为MT实际应用中的重要组成部分，这项任务已越来越受到关注。在本文中，我们首先提出了XLMRScore，这是一种基于使用XLM-Roberta（XLMR）模型计算的BertScore的简单无监督的QE方法，同时讨论了使用此方法发生的问题。接下来，我们建议两种减轻问题的方法：用未知令牌和预训练模型的跨语性对准替换未翻译的单词，以表示彼此之间的一致性单词。我们在WMT21 QE共享任务的四个低资源语言对上评估了所提出的方法，以及本文介绍的新的英语FARSI测试数据集。实验表明，我们的方法可以在两个零射击方案的监督基线中获得可比的结果，即皮尔森相关性的差异少于0.01，同时在所有低资源语言对中的平均低资源语言对中的无人看管竞争对手的平均水平超过8％的平均水平超过8％。 。

神经机器翻译（NMT）是一个开放的词汇问题。结果，处理在培训期间没有出现的单词（又称唱歌外（OOV）单词）长期以来一直是NMT系统的基本挑战。解决此问题的主要方法是字节对编码（BPE），将包括OOV单词在内的单词分为子字段中。在自动评估指标方面，BPE为广泛的翻译任务取得了令人印象深刻的结果。尽管通常假定使用BPE，但NMT系统能够处理OOV单词，但BPE在翻译OOV单词中的有效性尚未明确测量。在本文中，我们研究了BPE在多大程度上成功地翻译了单词级别的OOV单词。我们根据单词类型，段数，交叉注意权重和训练数据中段NGram的段频率分析OOV单词的翻译质量。我们的实验表明，尽管仔细的BPE设置似乎在整个数据集中翻译OOV单词时相当有用，但很大一部分的OOV单词被错误地翻译而成。此外，我们强调了BPE在为特殊案例（例如命名本性和涉及的语言彼此接近的语言）翻译OOV单词中的有效性稍高。

注释数据是用于培训和评估机器学习模型的自然语言处理中的重要成分。因此，注释具有高质量是非常理想的。但是，最近的工作表明，几个流行的数据集包含令人惊讶的注释错误或不一致之处。为了减轻此问题，多年来已经设计了许多注释错误检测方法。尽管研究人员表明他们的方法在新介绍的数据集上效果很好，但他们很少将其方法与以前的工作或同一数据集进行比较。这引起了人们对方法的一般表现的强烈关注，并且使他们的优势和劣势很难解决。因此，我们重新实现18种检测潜在注释错误的方法，并在9个英语数据集上对其进行评估，以进行文本分类以及令牌和跨度标签。此外，我们定义了统一的评估设置，包括注释错误检测任务，评估协议和一般最佳实践的新形式化。为了促进未来的研究和可重复性，我们将数据集和实施释放到易于使用和开源软件包中。

我们提出了两种小型无监督方法，用于消除文本中的毒性。我们的第一个方法结合了最近的两个想法：（1）使用小型条件语言模型的生成过程的指导和（2）使用释义模型进行风格传输。我们使用良好的令人措辞的令人愉快的释放器，由风格培训的语言模型引导，以保持文本内容并消除毒性。我们的第二种方法使用BERT用他们的非攻击性同义词取代毒性单词。我们通过使BERT替换具有可变数量的单词的屏蔽令牌来使该方法更灵活。最后，我们介绍了毒性去除任务的风格转移模型的第一个大规模比较研究。我们将模型与许多用于样式传输的方法进行比较。使用无监督的样式传输指标的组合以可参考方式评估该模型。两种方法都建议产生新的SOTA结果。

Reranking methods in machine translation aim to close the gap between common evaluation metrics (e.g. BLEU) and maximum likelihood learning and decoding algorithms. Prior works address this challenge by training models to rerank beam search candidates according to their predicted BLEU scores, building upon large models pretrained on massive monolingual corpora -- a privilege that was never made available to the baseline translation model. In this work, we examine a simple approach for training rerankers to predict translation candidates' BLEU scores without introducing additional data or parameters. Our approach can be used as a clean baseline, decoupled from external factors, for future research in this area.