Subword units are an effective way to alleviate the open vocabulary problems in neural machine translation (NMT). While sentences are usually converted into unique subword sequences, subword segmentation is potentially ambiguous and multiple segmentations are possible even with the same vocabulary. The question addressed in this paper is whether it is possible to harness the segmentation ambiguity as a noise to improve the robustness of NMT. We present a simple regularization method, subword regularization, which trains the model with multiple subword segmentations probabilistically sampled during training. In addition, for better subword sampling, we propose a new subword segmentation algorithm based on a unigram language model. We experiment with multiple corpora and report consistent improvements especially on low resource and out-of-domain settings.

This paper describes SentencePiece, a language-independent subword tokenizer and detokenizer designed for Neural-based text processing, including Neural Machine Translation. It provides open-source C++ and Python implementations for subword units. While existing subword segmentation tools assume that the input is pre-tokenized into word sequences, SentencePiece can train subword models directly from raw sentences, which allows us to make a purely end-to-end and language independent system. We perform a validation experiment of NMT on English-Japanese machine translation, and find that it is possible to achieve comparable accuracy to direct subword training from raw sentences. We also compare the performance of subword training and segmentation with various configurations. SentencePiece is available under the Apache 2 license at https://github.com/google/ sentencepiece.

Neural machine translation (NMT) models typically operate with a fixed vocabulary, but translation is an open-vocabulary problem. Previous work addresses the translation of out-of-vocabulary words by backing off to a dictionary. In this paper, we introduce a simpler and more effective approach, making the NMT model capable of open-vocabulary translation by encoding rare and unknown words as sequences of subword units. This is based on the intuition that various word classes are translatable via smaller units than words, for instance names (via character copying or transliteration), compounds (via compositional translation), and cognates and loanwords (via phonological and morphological transformations). We discuss the suitability of different word segmentation techniques, including simple character ngram models and a segmentation based on the byte pair encoding compression algorithm, and empirically show that subword models improve over a back-off dictionary baseline for the WMT 15 translation tasks English→German and English→Russian by up to 1.1 and 1.3 BLEU, respectively.

Neural Machine Translation (NMT) is an end-to-end learning approach for automated translation, with the potential to overcome many of the weaknesses of conventional phrase-based translation systems. Unfortunately, NMT systems are known to be computationally expensive both in training and in translation inference -sometimes prohibitively so in the case of very large data sets and large models. Several authors have also charged that NMT systems lack robustness, particularly when input sentences contain rare words. These issues have hindered NMT's use in practical deployments and services, where both accuracy and speed are essential. In this work, we present GNMT, Google's Neural Machine Translation system, which attempts to address many of these issues. Our model consists of a deep LSTM network with 8 encoder and 8 decoder layers using residual connections as well as attention connections from the decoder network to the encoder. To improve parallelism and therefore decrease training time, our attention mechanism connects the bottom layer of the decoder to the top layer of the encoder. To accelerate the final translation speed, we employ low-precision arithmetic during inference computations. To improve handling of rare words, we divide words into a limited set of common sub-word units ("wordpieces") for both input and output. This method provides a good balance between the flexibility of "character"-delimited models and the efficiency of "word"-delimited models, naturally handles translation of rare words, and ultimately improves the overall accuracy of the system. Our beam search technique employs a length-normalization procedure and uses a coverage penalty, which encourages generation of an output sentence that is most likely to cover all the words in the source sentence. To directly optimize the translation BLEU scores, we consider refining the models by using reinforcement learning, but we found that the improvement in the BLEU scores did not reflect in the human evaluation. On the WMT'14 English-to-French and English-to-German benchmarks, GNMT achieves competitive results to state-of-the-art. Using a human side-by-side evaluation on a set of isolated simple sentences, it reduces translation errors by an average of 60% compared to Google's phrase-based production system.

我们想要模型的文本单位是什么？从字节到多字表达式，可以在许多粒度下分析和生成文本。直到最近，大多数自然语言处理（NLP）模型通过单词操作，将那些作为离散和原子令牌处理，但从字节对编码（BPE）开始，基于次字的方法在许多领域都变得占主导地位，使得仍然存在小词汇表允许快速推断。是道路字符级模型的结束或字节级处理吗？在这项调查中，我们通过展示和评估基于学习分割的词语和字符以及基于子字的方法的混合方法以及基于学习的分割的杂交方法，连接多行工作。我们得出结论，对于所有应用来说，并且可能永远不会成为所有应用的银子弹奇异解决方案，并且严重思考令牌化对许多应用仍然很重要。

Multilingual pretrained models are effective for machine translation and cross-lingual processing because they contain multiple languages in one model. However, they are pretrained after their tokenizers are fixed; therefore it is difficult to change the vocabulary after pretraining. When we extend the pretrained models to new languages, we must modify the tokenizers simultaneously. In this paper, we add new subwords to the SentencePiece tokenizer to apply a multilingual pretrained model to new languages (Inuktitut in this paper). In our experiments, we segmented Inuktitut sentences into subwords without changing the segmentation of already pretrained languages, and applied the mBART-50 pretrained model to English-Inuktitut translation.

Neural Machine Translation (NMT) has obtained state-of-the art performance for several language pairs, while only using parallel data for training. Targetside monolingual data plays an important role in boosting fluency for phrasebased statistical machine translation, and we investigate the use of monolingual data for NMT. In contrast to previous work, which combines NMT models with separately trained language models, we note that encoder-decoder NMT architectures already have the capacity to learn the same information as a language model, and we explore strategies to train with monolingual data without changing the neural network architecture. By pairing monolingual training data with an automatic backtranslation, we can treat it as additional parallel training data, and we obtain substantial improvements on the WMT 15 task English↔German (+2.8-3.7 BLEU), and for the low-resourced IWSLT 14 task Turkish→English (+2.1-3.4 BLEU), obtaining new state-of-the-art results. We also show that fine-tuning on in-domain monolingual and parallel data gives substantial improvements for the IWSLT 15 task English→German.

令牌词汇的选择会影响机器翻译的性能。本文旨在弄清楚什么是良好的词汇，以及没有试用培训的最佳词汇。为了回答这些问题，我们首先为从信息理论的角度提供了对词汇的作用的替代理解。这是由此激励，我们制定了词汇化的追求 - 找到了具有正确尺寸的最佳令牌词典 - 作为最佳运输（OT）问题。我们提出Volt，简单而有效的解决方案，没有试用培训。经验结果表明，在不同场景中，Volt优于广泛使用的词汇，包括WMT-14英语 - 德语和TED的52翻译方向。例如，伏特达到近70％的词汇量减少和英语 - 德语翻译中的0.5个BLEU增益。此外，与BPE搜索相比，Volt在英语 - 德语翻译中将搜索时间从384 GPU小时从384 GPU小时到30 GPU小时。代码在https://github.com/jingjing-nlp/volt上获得。

关于阿塞拜疆的神经机器翻译（NMT）的研究很少。在本文中，我们将阿塞拜疆 - 英语NMT系统的性能基于一系列技术和数据集的性能。我们评估哪种细分技术在阿塞拜疆翻译上最有效，并基准了阿塞拜疆NMT模型在几个文本领域中的性能。我们的结果表明，虽然Umigram细分改善了NMT的性能，而Azerbaijani翻译模型则比数量更好，但跨域泛化仍然是一个挑战

The machine translation mechanism translates texts automatically between different natural languages, and Neural Machine Translation (NMT) has gained attention for its rational context analysis and fluent translation accuracy. However, processing low-resource languages that lack relevant training attributes like supervised data is a current challenge for Natural Language Processing (NLP). We incorporated a technique known Active Learning with the NMT toolkit Joey NMT to reach sufficient accuracy and robust predictions of low-resource language translation. With active learning, a semi-supervised machine learning strategy, the training algorithm determines which unlabeled data would be the most beneficial for obtaining labels using selected query techniques. We implemented two model-driven acquisition functions for selecting the samples to be validated. This work uses transformer-based NMT systems; baseline model (BM), fully trained model (FTM) , active learning least confidence based model (ALLCM), and active learning margin sampling based model (ALMSM) when translating English to Hindi. The Bilingual Evaluation Understudy (BLEU) metric has been used to evaluate system results. The BLEU scores of BM, FTM, ALLCM and ALMSM systems are 16.26, 22.56 , 24.54, and 24.20, respectively. The findings in this paper demonstrate that active learning techniques helps the model to converge early and improve the overall quality of the translation system.

本文介绍了一种新的数据增强方法，用于神经机器翻译，该方法可以在语言内部和跨语言内部实施更强的语义一致性。我们的方法基于条件掩盖语言模型（CMLM），该模型是双向的，可以在左右上下文以及标签上有条件。我们证明CMLM是生成上下文依赖性单词分布的好技术。特别是，我们表明CMLM能够通过在替换过程中对源和目标进行调节来实现语义一致性。此外，为了增强多样性，我们将软词替换的想法纳入了数据增强，该概念用词汇上的概率分布代替了一个单词。在不同量表的四个翻译数据集上进行的实验表明，总体解决方案会导致更现实的数据增强和更好的翻译质量。与最新作品相比，我们的方法始终取得了最佳性能，并且在基线上的提高了1.90个BLEU点。

Pre-training is an effective technique for ensuring robust performance on a variety of machine learning tasks. It typically depends on large-scale crawled corpora that can result in toxic or biased models. Such data can also be problematic with respect to copyright, attribution, and privacy. Pre-training with synthetic tasks and data is a promising way of alleviating such concerns since no real-world information is ingested by the model. Our goal in this paper is to understand what makes for a good pre-trained model when using synthetic resources. We answer this question in the context of neural machine translation by considering two novel approaches to translation model pre-training. Our first approach studies the effect of pre-training on obfuscated data derived from a parallel corpus by mapping words to a vocabulary of 'nonsense' tokens. Our second approach explores the effect of pre-training on procedurally generated synthetic parallel data that does not depend on any real human language corpus. Our empirical evaluation on multiple language pairs shows that, to a surprising degree, the benefits of pre-training can be realized even with obfuscated or purely synthetic parallel data. In our analysis, we consider the extent to which obfuscated and synthetic pre-training techniques can be used to mitigate the issue of hallucinated model toxicity.

随着自然语言处理领域的最新发展，在使用不同架构的神经机翻译中的使用情况上升了。变压器架构用于实现最先进的准确性，但它们是训练的非常昂贵的昂贵。每个人都不能拥有由高端GPU和其他资源组成的等待。我们在低计算资源上培训我们的模型，并调查结果。正如预期的那样，变形金刚表现出其他架构，但结果有一些令人惊讶的结果。由更多编码器和解码器组成的变形金刚需要花更多的时间来训练，但有更少的BLEU分数。LSTM在实验中表现良好，比较少花时间训练而不是变压器，适合在具有时间限制的情况下使用。

The word alignment task, despite its prominence in the era of statistical machine translation (SMT), is niche and under-explored today. In this two-part tutorial, we argue for the continued relevance for word alignment. The first part provides a historical background to word alignment as a core component of the traditional SMT pipeline. We zero-in on GIZA++, an unsupervised, statistical word aligner with surprising longevity. Jumping forward to the era of neural machine translation (NMT), we show how insights from word alignment inspired the attention mechanism fundamental to present-day NMT. The second part shifts to a survey approach. We cover neural word aligners, showing the slow but steady progress towards surpassing GIZA++ performance. Finally, we cover the present-day applications of word alignment, from cross-lingual annotation projection, to improving translation.

We propose a simple solution to use a single Neural Machine Translation (NMT) model to translate between multiple languages. Our solution requires no changes to the model architecture from a standard NMT system but instead introduces an artificial token at the beginning of the input sentence to specify the required target language. The rest of the model, which includes an encoder, decoder and attention module, remains unchanged and is shared across all languages. Using a shared wordpiece vocabulary, our approach enables Multilingual NMT using a single model without any increase in parameters, which is significantly simpler than previous proposals for Multilingual NMT. On the WMT'14 benchmarks, a single multilingual model achieves comparable performance for English→French and surpasses state-of-the-art results for English→German. Similarly, a single multilingual model surpasses state-of-the-art results for French→English and German→English on WMT'14 and WMT'15 benchmarks, respectively. On production corpora, multilingual models of up to twelve language pairs allow for better translation of many individual pairs. In addition to improving the translation quality of language pairs that the model was trained with, our models can also learn to perform implicit bridging between language pairs never seen explicitly during training, showing that transfer learning and zero-shot translation is possible for neural translation. Finally, we show analyses that hints at a universal interlingua representation in our models and show some interesting examples when mixing languages.

In this paper, we study the use of deep Transformer translation model for the CCMT 2022 Chinese-Thai low-resource machine translation task. We first explore the experiment settings (including the number of BPE merge operations, dropout probability, embedding size, etc.) for the low-resource scenario with the 6-layer Transformer. Considering that increasing the number of layers also increases the regularization on new model parameters (dropout modules are also introduced when using more layers), we adopt the highest performance setting but increase the depth of the Transformer to 24 layers to obtain improved translation quality. Our work obtains the SOTA performance in the Chinese-to-Thai translation in the constrained evaluation.

机器翻译系统（MTS）是通过将文本或语音从一种语言转换为另一种语言的有效工具。在像印度这样的大型多语言环境中，对有效的翻译系统的需求变得显而易见，英语和一套印度语言（ILS）正式使用。与英语相反，由于语料库的不可用，IL仍然被视为低资源语言。为了解决不对称性质，多语言神经机器翻译（MNMT）系统会发展为在这个方向上的理想方法。在本文中，我们提出了一个MNMT系统，以解决与低资源语言翻译有关的问题。我们的模型包括两个MNMT系统，即用于英语印度（一对多），另一个用于指示英语（多一对多），其中包含15个语言对（30个翻译说明）的共享编码器码头。由于大多数IL对具有很少的平行语料库，因此不足以训练任何机器翻译模型。我们探索各种增强策略，以通过建议的模型提高整体翻译质量。最先进的变压器体系结构用于实现所提出的模型。大量数据的试验揭示了其优越性比常规模型的优势。此外，本文解决了语言关系的使用（在方言，脚本等方面），尤其是关于同一家族的高资源语言在提高低资源语言表现方面的作用。此外，实验结果还表明了ILS的倒退和域适应性的优势，以提高源和目标语言的翻译质量。使用所有这些关键方法，我们提出的模型在评估指标方面比基线模型更有效，即一组ILS的BLEU（双语评估研究）得分。

This paper demonstrates that multilingual denoising pre-training produces significant performance gains across a wide variety of machine translation (MT) tasks. We present mBART -a sequence-to-sequence denoising auto-encoder pre-trained on large-scale monolingual corpora in many languages using the BART objective . mBART is the first method for pre-training a complete sequence-to-sequence model by denoising full texts in multiple languages, while previous approaches have focused only on the encoder, decoder, or reconstructing parts of the text. Pre-training a complete model allows it to be directly fine tuned for supervised (both sentence-level and document-level) and unsupervised machine translation, with no task-specific modifications. We demonstrate that adding mBART initialization produces performance gains in all but the highest-resource settings, including up to 12 BLEU points for low resource MT and over 5 BLEU points for many document-level and unsupervised models. We also show it also enables new types of transfer to language pairs with no bi-text or that were not in the pre-training corpus, and present extensive analysis of which factors contribute the most to effective pre-training.

我们介绍了双图：一种简单但有效的训练策略，以提高神经机器翻译（NMT）性能。它由两个程序组成：双向预处理和单向填充。这两个过程均使用SIMCUT，这是一种简单的正则化方法，迫使原始句子对的输出分布之间的一致性。在不利用额外的数据集通过反翻译或集成大规模预认证的模型的情况下，BI-Simcut可以在五个翻译基准（数据尺寸从160K到20.20万）中实现强大的翻译性能：EN-的BLEU得分为31.16，EN-> DE和38.37的BLEU得分为38.37 de-> en在IWSLT14数据集上，en-> de的30.78和35.15在WMT14数据集上进行DE-> en，而WMT17数据集中的ZH-> EN为27.17。 Simcut不是一种新方法，而是简化和适用于NMT的cutoff（Shen等，2020）的版本，可以将其视为基于扰动的方法。鉴于Simcut和Bi-Simcut的普遍性和简单性，我们认为它们可以作为未来NMT研究的强大基准。

神经机器翻译（NMT）模型在大型双语数据集上已有效。但是，现有的方法和技术表明，该模型的性能高度取决于培训数据中的示例数量。对于许多语言而言，拥有如此数量的语料库是一个牵强的梦想。我们从单语言词典探索新语言的单语扬声器中汲取灵感，我们研究了双语词典对具有极低或双语语料库的语言的适用性。在本文中，我们使用具有NMT模型的双语词典探索方法，以改善资源极低的资源语言的翻译。我们将此工作扩展到多语言系统，表现出零拍的属性。我们详细介绍了字典质量，培训数据集大小，语言家族等对翻译质量的影响。多种低资源测试语言的结果表明，我们的双语词典方法比基线相比。