令牌词汇的选择会影响机器翻译的性能。本文旨在弄清楚什么是良好的词汇，以及没有试用培训的最佳词汇。为了回答这些问题，我们首先为从信息理论的角度提供了对词汇的作用的替代理解。这是由此激励，我们制定了词汇化的追求 - 找到了具有正确尺寸的最佳令牌词典 - 作为最佳运输（OT）问题。我们提出Volt，简单而有效的解决方案，没有试用培训。经验结果表明，在不同场景中，Volt优于广泛使用的词汇，包括WMT-14英语 - 德语和TED的52翻译方向。例如，伏特达到近70％的词汇量减少和英语 - 德语翻译中的0.5个BLEU增益。此外，与BPE搜索相比，Volt在英语 - 德语翻译中将搜索时间从384 GPU小时从384 GPU小时到30 GPU小时。代码在https://github.com/jingjing-nlp/volt上获得。

Subword units are an effective way to alleviate the open vocabulary problems in neural machine translation (NMT). While sentences are usually converted into unique subword sequences, subword segmentation is potentially ambiguous and multiple segmentations are possible even with the same vocabulary. The question addressed in this paper is whether it is possible to harness the segmentation ambiguity as a noise to improve the robustness of NMT. We present a simple regularization method, subword regularization, which trains the model with multiple subword segmentations probabilistically sampled during training. In addition, for better subword sampling, we propose a new subword segmentation algorithm based on a unigram language model. We experiment with multiple corpora and report consistent improvements especially on low resource and out-of-domain settings.

多语言代币器是多语言神经机器翻译的基本组成部分。它是通过多语种语料库训练的。由于偏斜的数据分布被认为是有害的，因此通常使用采样策略来平衡语料库中的语言。但是，很少有作品系统地回答了令牌训练中的语言失衡如何影响下游的表现。在这项工作中，我们分析了翻译性能如何随着语言之间的数据比率而变化，而在令牌培训语料库中的变化。我们发现，虽然当语言更加同样地采样时，通常会观察到相对较好的性能，但下游性能对语言不平衡的性能比我们通常预期的要强。在执行任务之前，可以警告两个功能，即UNK速率和接近角色水平，可以警告下游性能不佳。我们还将令牌训练的语言抽样与模型培训的采样分开，并表明该模型对后者更敏感。

Pre-training is an effective technique for ensuring robust performance on a variety of machine learning tasks. It typically depends on large-scale crawled corpora that can result in toxic or biased models. Such data can also be problematic with respect to copyright, attribution, and privacy. Pre-training with synthetic tasks and data is a promising way of alleviating such concerns since no real-world information is ingested by the model. Our goal in this paper is to understand what makes for a good pre-trained model when using synthetic resources. We answer this question in the context of neural machine translation by considering two novel approaches to translation model pre-training. Our first approach studies the effect of pre-training on obfuscated data derived from a parallel corpus by mapping words to a vocabulary of 'nonsense' tokens. Our second approach explores the effect of pre-training on procedurally generated synthetic parallel data that does not depend on any real human language corpus. Our empirical evaluation on multiple language pairs shows that, to a surprising degree, the benefits of pre-training can be realized even with obfuscated or purely synthetic parallel data. In our analysis, we consider the extent to which obfuscated and synthetic pre-training techniques can be used to mitigate the issue of hallucinated model toxicity.

Neural machine translation (NMT) models typically operate with a fixed vocabulary, but translation is an open-vocabulary problem. Previous work addresses the translation of out-of-vocabulary words by backing off to a dictionary. In this paper, we introduce a simpler and more effective approach, making the NMT model capable of open-vocabulary translation by encoding rare and unknown words as sequences of subword units. This is based on the intuition that various word classes are translatable via smaller units than words, for instance names (via character copying or transliteration), compounds (via compositional translation), and cognates and loanwords (via phonological and morphological transformations). We discuss the suitability of different word segmentation techniques, including simple character ngram models and a segmentation based on the byte pair encoding compression algorithm, and empirically show that subword models improve over a back-off dictionary baseline for the WMT 15 translation tasks English→German and English→Russian by up to 1.1 and 1.3 BLEU, respectively.

多语种NMT已成为MT在生产中部署的有吸引力的解决方案。但是要匹配双语质量，它符合较大且较慢的型号。在这项工作中，我们考虑了几种方法在推理时更快地使多语言NMT变得更快而不会降低其质量。我们在两种20语言多平行设置中尝试几个“光解码器”架构：在TED会谈中小规模和帕拉克曲线上的大规模。我们的实验表明，将具有词汇过滤的浅解码器组合在于，在翻译质量下没有损失的速度超过两倍。我们用Bleu和Chrf（380语言对），鲁棒性评估和人类评估验证了我们的研究结果。

Minimum Bayesian Risk Decoding (MBR) emerges as a promising decoding algorithm in Neural Machine Translation. However, MBR performs poorly with label smoothing, which is surprising as label smoothing provides decent improvement with beam search and improves generality in various tasks. In this work, we show that the issue arises from the un-consistency of label smoothing on the token-level and sequence-level distributions. We demonstrate that even though label smoothing only causes a slight change in the token-level, the sequence-level distribution is highly skewed. We coin the issue \emph{distributional over-smoothness}. To address this issue, we propose a simple and effective method, Distributional Cooling MBR (DC-MBR), which manipulates the entropy of output distributions by tuning down the Softmax temperature. We theoretically prove the equivalence between pre-tuning label smoothing factor and distributional cooling. Experiments on NMT benchmarks validate that distributional cooling improves MBR's efficiency and effectiveness in various settings.

Multilingual pretrained models are effective for machine translation and cross-lingual processing because they contain multiple languages in one model. However, they are pretrained after their tokenizers are fixed; therefore it is difficult to change the vocabulary after pretraining. When we extend the pretrained models to new languages, we must modify the tokenizers simultaneously. In this paper, we add new subwords to the SentencePiece tokenizer to apply a multilingual pretrained model to new languages (Inuktitut in this paper). In our experiments, we segmented Inuktitut sentences into subwords without changing the segmentation of already pretrained languages, and applied the mBART-50 pretrained model to English-Inuktitut translation.

In this paper, we study the use of deep Transformer translation model for the CCMT 2022 Chinese-Thai low-resource machine translation task. We first explore the experiment settings (including the number of BPE merge operations, dropout probability, embedding size, etc.) for the low-resource scenario with the 6-layer Transformer. Considering that increasing the number of layers also increases the regularization on new model parameters (dropout modules are also introduced when using more layers), we adopt the highest performance setting but increase the depth of the Transformer to 24 layers to obtain improved translation quality. Our work obtains the SOTA performance in the Chinese-to-Thai translation in the constrained evaluation.

在几乎所有文本生成应用中，Word序列在左右（L2R）或左右（R2L）方式中构造，因为自然语言句子是写入L2R或R2L。但是，我们发现自然语言书面订单对文本生成至关重要。在本文中，我们提出了一种螺旋语言建模（SLM），这是一种普遍的方法，使人们能够构建超出L2R和R2L订单的自然语言句子。 SLM允许其中一个从结果文本内的任意令牌开始，并在所选的任意令牌中展开REST令牌。它使解码顺序除了语言模型困惑之外的新优化目标，这进一步提高了所生成文本的分集和质量。此外，SLM使得可以通过选择正确的开始令牌来操纵文本构建过程。 SLM还将生成排序引入了额外的正则化，以提高低资源方案中的模型稳健性。 8次广泛研究的神经机翻译（NMT）任务的实验表明，与传统的L2R解码方法相比，SLM高达4.7 BLEU增加。

非自动性变压器（NAT）是文本生成模型的家族，旨在通过并行预测整个句子来减少解码延迟。但是，这种延迟减少牺牲了捕获从左到右的依赖性的能力，从而使NAT学习非常具有挑战性。在本文中，我们介绍了理论和经验分析，以揭示NAT学习的挑战，并提出统一的观点来了解现有的成功。首先，我们表明，简单地通过最大化可能性来训练NAT可以导致边际分布的近似值，但在代币之间降低了所有依赖关系，在该数据集的条件总相关性可以测量删除的信息。其次，我们在统一的框架中正式化了许多以前的目标，并表明他们的成功可以得出结论，以最大程度地提高代理分布的可能性，从而减少了信息损失。实证研究表明，我们的观点可以解释NAT学习中的现象，并指导新培训方法的设计。

我们介绍了双图：一种简单但有效的训练策略，以提高神经机器翻译（NMT）性能。它由两个程序组成：双向预处理和单向填充。这两个过程均使用SIMCUT，这是一种简单的正则化方法，迫使原始句子对的输出分布之间的一致性。在不利用额外的数据集通过反翻译或集成大规模预认证的模型的情况下，BI-Simcut可以在五个翻译基准（数据尺寸从160K到20.20万）中实现强大的翻译性能：EN-的BLEU得分为31.16，EN-> DE和38.37的BLEU得分为38.37 de-> en在IWSLT14数据集上，en-> de的30.78和35.15在WMT14数据集上进行DE-> en，而WMT17数据集中的ZH-> EN为27.17。 Simcut不是一种新方法，而是简化和适用于NMT的cutoff（Shen等，2020）的版本，可以将其视为基于扰动的方法。鉴于Simcut和Bi-Simcut的普遍性和简单性，我们认为它们可以作为未来NMT研究的强大基准。

The word alignment task, despite its prominence in the era of statistical machine translation (SMT), is niche and under-explored today. In this two-part tutorial, we argue for the continued relevance for word alignment. The first part provides a historical background to word alignment as a core component of the traditional SMT pipeline. We zero-in on GIZA++, an unsupervised, statistical word aligner with surprising longevity. Jumping forward to the era of neural machine translation (NMT), we show how insights from word alignment inspired the attention mechanism fundamental to present-day NMT. The second part shifts to a survey approach. We cover neural word aligners, showing the slow but steady progress towards surpassing GIZA++ performance. Finally, we cover the present-day applications of word alignment, from cross-lingual annotation projection, to improving translation.

神经文本生成模型，如用于总结和翻译的那些模型产生高质量的输出，但是当我们真正想要的是一个不同的选项时，通常会集中在模式周围。我们介绍了一个搜索算法来构建编码大量生成选项的格子。首先，我们将解码重组为最佳搜索，该搜索探讨了与光束搜索不同的空间，并通过避免修剪路径来提高效率。其次，我们重新审视假设重组的想法：我们可以在搜索期间识别类似的生成候选者，并将它们合并为近似。在摘要和机器翻译中，我们表明我们的算法编码了数百到数千个不同的选项，这些选项保持语法和高质量成一个线性型格子。该算法为在大规模不同输出之上构建下游生成应用提供了基础。

我们想要模型的文本单位是什么？从字节到多字表达式，可以在许多粒度下分析和生成文本。直到最近，大多数自然语言处理（NLP）模型通过单词操作，将那些作为离散和原子令牌处理，但从字节对编码（BPE）开始，基于次字的方法在许多领域都变得占主导地位，使得仍然存在小词汇表允许快速推断。是道路字符级模型的结束或字节级处理吗？在这项调查中，我们通过展示和评估基于学习分割的词语和字符以及基于子字的方法的混合方法以及基于学习的分割的杂交方法，连接多行工作。我们得出结论，对于所有应用来说，并且可能永远不会成为所有应用的银子弹奇异解决方案，并且严重思考令牌化对许多应用仍然很重要。

关于阿塞拜疆的神经机器翻译（NMT）的研究很少。在本文中，我们将阿塞拜疆 - 英语NMT系统的性能基于一系列技术和数据集的性能。我们评估哪种细分技术在阿塞拜疆翻译上最有效，并基准了阿塞拜疆NMT模型在几个文本领域中的性能。我们的结果表明，虽然Umigram细分改善了NMT的性能，而Azerbaijani翻译模型则比数量更好，但跨域泛化仍然是一个挑战

不断增长的数据量导致更大的通用模型。通常遗漏特定用例，因为通用模型在域特定情况下往往表现不佳。我们的工作通过用于从通用域（并行文本）语料库的域名数据的方法解决了这个差距，用于机器翻译的任务。所提出的方法根据具有单孔域的特定数据集的余弦相似度在并行通用域数据中排列句子。然后，我们选择具有最高相似性分数的顶级k句，以培训调整的新机器翻译系统到特定的域数据。我们的实验结果表明，在通用或通用和域数据的混合训练的域内训练的模型训练的模型。也就是说，我们的方法以低计算成本和数据大小选择高质量的域特定培训实例。

This paper demonstrates that multilingual denoising pre-training produces significant performance gains across a wide variety of machine translation (MT) tasks. We present mBART -a sequence-to-sequence denoising auto-encoder pre-trained on large-scale monolingual corpora in many languages using the BART objective . mBART is the first method for pre-training a complete sequence-to-sequence model by denoising full texts in multiple languages, while previous approaches have focused only on the encoder, decoder, or reconstructing parts of the text. Pre-training a complete model allows it to be directly fine tuned for supervised (both sentence-level and document-level) and unsupervised machine translation, with no task-specific modifications. We demonstrate that adding mBART initialization produces performance gains in all but the highest-resource settings, including up to 12 BLEU points for low resource MT and over 5 BLEU points for many document-level and unsupervised models. We also show it also enables new types of transfer to language pairs with no bi-text or that were not in the pre-training corpus, and present extensive analysis of which factors contribute the most to effective pre-training.

Subword-level models have been the dominant paradigm in NLP. However, character-level models have the benefit of seeing each character individually, providing the model with more detailed information that ultimately could lead to better models. Recent works have shown character-level models to be competitive with subword models, but costly in terms of time and computation. Character-level models with a downsampling component alleviate this, but at the cost of quality, particularly for machine translation. This work analyzes the problems of previous downsampling methods and introduces a novel downsampling method which is informed by subwords. This new downsampling method not only outperforms existing downsampling methods, showing that downsampling characters can be done without sacrificing quality, but also leads to promising performance compared to subword models for translation.

低频词预测仍然是现代神经电机翻译（NMT）系统的挑战。最近的自适应培训方法通过强调整体培训目标的重量来促进不频繁词语的产出。尽管召回了低频词的召回，但它们的预测精度意外地受到自适应目标的阻碍。灵感来自观察到低频词形成更紧凑的嵌入空间，我们从代表学习角度解决这一挑战。具体地，我们提出了一种频率感知的令牌级对比度学习方法，其中每个解码步骤的隐藏状态以基于相应的字频率的柔和对比方式从其他目标单词的对应物推开。我们对广泛使用的NIST汉语 - 英语和WMT14英语 - 德语翻译任务进行实验。经验结果表明，我们的提出方法不仅可以显着提高翻译质量，还可以提高词汇分集和优化词表示空间。进一步调查揭示了，与相关的自适应培训策略相比，我们对低频词预测方法的优势在于在不牺牲精度的情况下在不同频率上的令牌级召回的鲁棒性。