这项工作适用于最低贝叶斯风险（MBR）解码，以优化翻译质量的各种自动化指标。机器翻译中的自动指标最近取得了巨大的进步。特别是，在人类评级（例如BLEurt，或Comet）上微调，在与人类判断的相关性方面是优于表面度量的微调。我们的实验表明，神经翻译模型与神经基于基于神经参考度量，BLEURT的组合导致自动和人类评估的显着改善。通过与经典光束搜索输出不同的翻译获得该改进：这些翻译的可能性较低，并且较少受到Bleu等表面度量的青睐。

神经指标与机器翻译系统评估中的人类判断达到了令人印象深刻的相关性，但是在我们可以安全地针对此类指标进行优化之前，我们应该意识到（并且理想地消除）偏向获得高分的不良翻译的偏见。我们的实验表明，基于样本的最小贝叶斯风险解码可用于探索和量化此类弱点。在将此策略应用于彗星进行ende和de-en时，我们发现彗星模型不足以差异和命名实体差异。我们进一步表明，通过简单地培训其他合成数据并发布我们的代码和数据以促进进一步的实验，这些偏见很难完全消除。

We propose BERTSCORE, an automatic evaluation metric for text generation. Analogously to common metrics, BERTSCORE computes a similarity score for each token in the candidate sentence with each token in the reference sentence. However, instead of exact matches, we compute token similarity using contextual embeddings. We evaluate using the outputs of 363 machine translation and image captioning systems. BERTSCORE correlates better with human judgments and provides stronger model selection performance than existing metrics. Finally, we use an adversarial paraphrase detection task to show that BERTSCORE is more robust to challenging examples when compared to existing metrics.

Reranking methods in machine translation aim to close the gap between common evaluation metrics (e.g. BLEU) and maximum likelihood learning and decoding algorithms. Prior works address this challenge by training models to rerank beam search candidates according to their predicted BLEU scores, building upon large models pretrained on massive monolingual corpora -- a privilege that was never made available to the baseline translation model. In this work, we examine a simple approach for training rerankers to predict translation candidates' BLEU scores without introducing additional data or parameters. Our approach can be used as a clean baseline, decoupled from external factors, for future research in this area.

Minimum Bayesian Risk Decoding (MBR) emerges as a promising decoding algorithm in Neural Machine Translation. However, MBR performs poorly with label smoothing, which is surprising as label smoothing provides decent improvement with beam search and improves generality in various tasks. In this work, we show that the issue arises from the un-consistency of label smoothing on the token-level and sequence-level distributions. We demonstrate that even though label smoothing only causes a slight change in the token-level, the sequence-level distribution is highly skewed. We coin the issue \emph{distributional over-smoothness}. To address this issue, we propose a simple and effective method, Distributional Cooling MBR (DC-MBR), which manipulates the entropy of output distributions by tuning down the Softmax temperature. We theoretically prove the equivalence between pre-tuning label smoothing factor and distributional cooling. Experiments on NMT benchmarks validate that distributional cooling improves MBR's efficiency and effectiveness in various settings.

As machine translation (MT) metrics improve their correlation with human judgement every year, it is crucial to understand the limitations of such metrics at the segment level. Specifically, it is important to investigate metric behaviour when facing accuracy errors in MT because these can have dangerous consequences in certain contexts (e.g., legal, medical). We curate ACES, a translation accuracy challenge set, consisting of 68 phenomena ranging from simple perturbations at the word/character level to more complex errors based on discourse and real-world knowledge. We use ACES to evaluate a wide range of MT metrics including the submissions to the WMT 2022 metrics shared task and perform several analyses leading to general recommendations for metric developers. We recommend: a) combining metrics with different strengths, b) developing metrics that give more weight to the source and less to surface-level overlap with the reference and c) explicitly modelling additional language-specific information beyond what is available via multilingual embeddings.

Large language models (LLMs) that have been trained on multilingual but not parallel text exhibit a remarkable ability to translate between languages. We probe this ability in an in-depth study of the pathways language model (PaLM), which has demonstrated the strongest machine translation (MT) performance among similarly-trained LLMs to date. We investigate various strategies for choosing translation examples for few-shot prompting, concluding that example quality is the most important factor. Using optimized prompts, we revisit previous assessments of PaLM's MT capabilities with more recent test sets, modern MT metrics, and human evaluation, and find that its performance, while impressive, still lags that of state-of-the-art supervised systems. We conclude by providing an analysis of PaLM's MT output which reveals some interesting properties and prospects for future work.

虽然已经提出了许多背景感知神经机器转换模型在翻译中包含语境，但大多数模型在句子级别对齐的并行文档上培训结束到底。因为只有少数域（和语言对）具有此类文档级并行数据，所以我们无法在大多数域中执行准确的上下文感知转换。因此，我们通过将文档级语言模型结合到解码器中，提出了一种简单的方法将句子级转换模型转换为上下文感知模型。我们的上下文感知解码器仅在句子级并行语料库和单语演模板上构建;因此，不需要文档级并行数据。在理论上，这项工作的核心部分是使用上下文和当前句子之间的点亮互信息的语境信息的新颖表示。我们以三种语言对，英语到法语，英语到俄语，以及日语到英语，通过评估，通过评估以及对上下文意识翻译的对比测试。

我们假设现有的句子级机器翻译（MT）指标在人类参考包含歧义时会效率降低。为了验证这一假设，我们提出了一种非常简单的方法，用于扩展预审计的指标以在文档级别合并上下文。我们将我们的方法应用于三个流行的指标，即Bertscore，Prism和Comet，以及无参考的公制Comet-QE。我们使用提供的MQM注释评估WMT 2021指标共享任务的扩展指标。我们的结果表明，扩展指标的表现在约85％的测试条件下优于其句子级别的级别，而在排除低质量人类参考的结果时。此外，我们表明我们的文档级扩展大大提高了其对话语现象任务的准确性，从而优于专用基线高达6.1％。我们的实验结果支持我们的初始假设，并表明对指标的简单扩展使他们能够利用上下文来解决参考中的歧义。

While the problem of hallucinations in neural machine translation has long been recognized, so far the progress on its alleviation is very little. Indeed, recently it turned out that without artificially encouraging models to hallucinate, previously existing methods fall short and even the standard sequence log-probability is more informative. It means that characteristics internal to the model can give much more information than we expect, and before using external models and measures, we first need to ask: how far can we go if we use nothing but the translation model itself ? We propose to use a method that evaluates the percentage of the source contribution to a generated translation. Intuitively, hallucinations are translations "detached" from the source, hence they can be identified by low source contribution. This method improves detection accuracy for the most severe hallucinations by a factor of 2 and is able to alleviate hallucinations at test time on par with the previous best approach that relies on external models. Next, if we move away from internal model characteristics and allow external tools, we show that using sentence similarity from cross-lingual embeddings further improves these results.

We present the CUNI-Bergamot submission for the WMT22 General translation task. We compete in English$\rightarrow$Czech direction. Our submission further explores block backtranslation techniques. Compared to the previous work, we measure performance in terms of COMET score and named entities translation accuracy. We evaluate performance of MBR decoding compared to traditional mixed backtranslation training and we show a possible synergy when using both of the techniques simultaneously. The results show that both approaches are effective means of improving translation quality and they yield even better results when combined.

本文介绍了亚当·米基维奇大学（Adam Mickiewicz University）（AMU）提交的《 WMT 2022一般MT任务》的踪迹。我们参加了乌克兰$ \ leftrightarrow $捷克翻译指示。这些系统是基于变压器（大）体系结构的四个模型的加权合奏。模型使用源因素来利用输入中存在的命名实体的信息。合奏中的每个模型仅使用共享任务组织者提供的数据培训。一种嘈杂的反向翻译技术用于增强培训语料库。合奏中的模型之一是文档级模型，该模型在平行和合成的更长序列上训练。在句子级的解码过程中，集合生成了N最佳列表。 n-最佳列表与单个文档级模型生成的n-最佳列表合并，该列表一次翻译了多个句子。最后，使用现有的质量估计模型和最小贝叶斯风险解码来重新列出N最好的列表，因此根据彗星评估指标选择了最佳假设。根据自动评估结果，我们的系统在两个翻译方向上排名第一。

End-to-End speech-to-speech translation (S2ST) is generally evaluated with text-based metrics. This means that generated speech has to be automatically transcribed, making the evaluation dependent on the availability and quality of automatic speech recognition (ASR) systems. In this paper, we propose a text-free evaluation metric for end-to-end S2ST, named BLASER, to avoid the dependency on ASR systems. BLASER leverages a multilingual multimodal encoder to directly encode the speech segments for source input, translation output and reference into a shared embedding space and computes a score of the translation quality that can be used as a proxy to human evaluation. To evaluate our approach, we construct training and evaluation sets from more than 40k human annotations covering seven language directions. The best results of BLASER are achieved by training with supervision from human rating scores. We show that when evaluated at the sentence level, BLASER correlates significantly better with human judgment compared to ASR-dependent metrics including ASR-SENTBLEU in all translation directions and ASR-COMET in five of them. Our analysis shows combining speech and text as inputs to BLASER does not increase the correlation with human scores, but best correlations are achieved when using speech, which motivates the goal of our research. Moreover, we show that using ASR for references is detrimental for text-based metrics.

Large-scale generative models show an impressive ability to perform a wide range of Natural Language Processing (NLP) tasks using in-context learning, where a few examples are used to describe a task to the model. For Machine Translation (MT), these examples are typically randomly sampled from the development dataset with a similar distribution as the evaluation set. However, it is unclear how the choice of these in-context examples and their ordering impacts the output translation quality. In this work, we aim to understand the properties of good in-context examples for MT in both in-domain and out-of-domain settings. We show that the translation quality and the domain of the in-context examples matter and that 1-shot noisy unrelated example can have a catastrophic impact on output quality. While concatenating multiple random examples reduces the effect of noise, a single good prompt optimized to maximize translation quality on the development dataset can elicit learned information from the pre-trained language model. Adding similar examples based on an n-gram overlap with the test source significantly and consistently improves the translation quality of the outputs, outperforming a strong kNN-MT baseline in 2 out of 4 out-of-domain datasets.

评估指标是文本生成系统的关键成分。近年来，已经提出了几十年前的文本生成质量的人类评估，提出了几个基于伯特的评估指标（包括Bertscore，Moverscore，BLEurt等），这些评估与文本生成质量的人类评估比Bleu或Rouge进行了更好。但是，很少是已知这些度量基于黑盒语言模型表示的指标实际捕获（通常假设它们模型语义相似性）。在这项工作中，我们使用基于简单的回归的全局解释技术来沿着语言因素解开度量标准分数，包括语义，语法，形态和词汇重叠。我们表明，不同的指标捕获了一定程度的各个方面，但它们对词汇重叠大大敏感，就像Bleu和Rouge一样。这暴露了这些新颖性拟议的指标的限制，我们还在对抗对抗测试场景中突出显示。

最近提出的基于BERT的评估指标在标准评估基准方面表现良好，但容易受到对抗性攻击的影响，例如与事实错误有关。我们认为这（部分原因）是因为它们是语义相似性的模型。相反，我们根据自然语言推断（NLI）制定评估指标，我们认为这是更合适的建模。我们设计了一个基于偏好的对抗攻击框架，并表明我们的基于NLI的指标比最近基于BERT的指标更强大。在标准基准上，我们的基于NLI的指标的表现优于现有的摘要指标，但在SOTA MT指标下执行。但是，当我们将现有指标与NLI指标相结合时，我们可以获得更高的对抗性鲁棒性（ +20％至 +30％）和较高质量的指标，如标准基准测量（ +5％至 +25％）。

我们介绍了一种新的分布式策略梯度算法，并表明它在优化机器翻译模型时，在培训稳定性和概括性绩效方面都优于现有的奖励感知培训程序，例如增强，最低风险培训（MRT）和近端政策优化（PPO）。我们称之为MAD的算法（由于在重要性加权计算中使用平均绝对偏差），它分布式数据生成器在Worker节点上每个源句子对多个候选者进行采样，而中心学习者则更新了策略。 MAD取决于两个降低差异策略：（1）一种有条件的奖励归一化方法，可确保每个源句子都具有正面和负面奖励翻译示例，以及（2）一种新的强大重要性加权方案，充当条件性熵正常化器。在各种翻译任务上进行的实验表明，使用MAD算法在使用贪婪的解码和梁搜索时，使用MAD算法学到的策略表现良好，并且学到的政策对训练过程中使用的特定奖励很敏感。

The word alignment task, despite its prominence in the era of statistical machine translation (SMT), is niche and under-explored today. In this two-part tutorial, we argue for the continued relevance for word alignment. The first part provides a historical background to word alignment as a core component of the traditional SMT pipeline. We zero-in on GIZA++, an unsupervised, statistical word aligner with surprising longevity. Jumping forward to the era of neural machine translation (NMT), we show how insights from word alignment inspired the attention mechanism fundamental to present-day NMT. The second part shifts to a survey approach. We cover neural word aligners, showing the slow but steady progress towards surpassing GIZA++ performance. Finally, we cover the present-day applications of word alignment, from cross-lingual annotation projection, to improving translation.

The machine translation mechanism translates texts automatically between different natural languages, and Neural Machine Translation (NMT) has gained attention for its rational context analysis and fluent translation accuracy. However, processing low-resource languages that lack relevant training attributes like supervised data is a current challenge for Natural Language Processing (NLP). We incorporated a technique known Active Learning with the NMT toolkit Joey NMT to reach sufficient accuracy and robust predictions of low-resource language translation. With active learning, a semi-supervised machine learning strategy, the training algorithm determines which unlabeled data would be the most beneficial for obtaining labels using selected query techniques. We implemented two model-driven acquisition functions for selecting the samples to be validated. This work uses transformer-based NMT systems; baseline model (BM), fully trained model (FTM) , active learning least confidence based model (ALLCM), and active learning margin sampling based model (ALMSM) when translating English to Hindi. The Bilingual Evaluation Understudy (BLEU) metric has been used to evaluate system results. The BLEU scores of BM, FTM, ALLCM and ALMSM systems are 16.26, 22.56 , 24.54, and 24.20, respectively. The findings in this paper demonstrate that active learning techniques helps the model to converge early and improve the overall quality of the translation system.

Text generation has made significant advances in the last few years. Yet, evaluation metrics have lagged behind, as the most popular choices (e.g., BLEU and ROUGE) may correlate poorly with human judgments. We propose BLEURT, a learned evaluation metric based on BERT that can model human judgments with a few thousand possibly biased training examples. A key aspect of our approach is a novel pre-training scheme that uses millions of synthetic examples to help the model generalize. BLEURT provides state-ofthe-art results on the last three years of the WMT Metrics shared task and the WebNLG Competition dataset. In contrast to a vanilla BERT-based approach, it yields superior results even when the training data is scarce and out-of-distribution.