本文提出了一种简单而有效的方法，可以改善两种情况下的直接（x-to-y）翻译：零射击和直接数据时。我们将编码器和解码器的输入令牌修改为包括源和目标语言的信号。我们在从头开始训练或使用拟议的设置对验证模型进行填充时显示出绩效增长。在实验中，根据检查点选择标准，我们的方法在内部数据集上显示了近10.0个BLEU点的增益。在WMT评估活动中，从英语性能提高了4.17和2.87 BLEU点，在零射击设置和直接数据可用于培训时。而X-to-y在零射基线上提高了1.29 BLEU，而在多到许多基线上提高了0.44。在低资源设置中，我们在X-TO-Y域数据上进行填充时会看到1.5〜1.7点的改善。

We propose a simple solution to use a single Neural Machine Translation (NMT) model to translate between multiple languages. Our solution requires no changes to the model architecture from a standard NMT system but instead introduces an artificial token at the beginning of the input sentence to specify the required target language. The rest of the model, which includes an encoder, decoder and attention module, remains unchanged and is shared across all languages. Using a shared wordpiece vocabulary, our approach enables Multilingual NMT using a single model without any increase in parameters, which is significantly simpler than previous proposals for Multilingual NMT. On the WMT'14 benchmarks, a single multilingual model achieves comparable performance for English→French and surpasses state-of-the-art results for English→German. Similarly, a single multilingual model surpasses state-of-the-art results for French→English and German→English on WMT'14 and WMT'15 benchmarks, respectively. On production corpora, multilingual models of up to twelve language pairs allow for better translation of many individual pairs. In addition to improving the translation quality of language pairs that the model was trained with, our models can also learn to perform implicit bridging between language pairs never seen explicitly during training, showing that transfer learning and zero-shot translation is possible for neural translation. Finally, we show analyses that hints at a universal interlingua representation in our models and show some interesting examples when mixing languages.

多语言神经机器翻译（MNMT）使一个系统能够将句子从多种源语言转换为多种目标语言，与传统的双语系统相比，大大降低了部署成本。但是，MNMT培训益处通常仅限于多一对一的方向。该模型在一对一的表现不佳，并且在零镜头设置中遭受了多种影响。为了解决这个问题，本文讨论了如何实际构建提供任意X-Y翻译指示的MNMT系统，同时使用预处理和填充的两阶段培训策略利用多语言。尝试WMT'21多语言翻译任务，我们证明我们的系统的表现优于大多数方向的直接双语模型和枢轴翻译模型的传统基线，平均提供+6.0和+4.1 BLEU，而无需进行架构更改或额外的数据收集。 。此外，我们还在极大的数据设置中检查了我们提出的方法，以适应实际的部署方案。

多语种NMT已成为MT在生产中部署的有吸引力的解决方案。但是要匹配双语质量，它符合较大且较慢的型号。在这项工作中，我们考虑了几种方法在推理时更快地使多语言NMT变得更快而不会降低其质量。我们在两种20语言多平行设置中尝试几个“光解码器”架构：在TED会谈中小规模和帕拉克曲线上的大规模。我们的实验表明，将具有词汇过滤的浅解码器组合在于，在翻译质量下没有损失的速度超过两倍。我们用Bleu和Chrf（380语言对），鲁棒性评估和人类评估验证了我们的研究结果。

我们提出了一种两阶段的培训方法，用于开发单个NMT模型，以翻译英语和英语的看不见的语言。对于第一阶段，我们将编码器模型初始化以鉴定XLM-R和Roberta的权重，然后对25种语言的平行数据进行多种语言微调。我们发现该模型可以推广到对看不见的语言的零击翻译。在第二阶段，我们利用这种概括能力从单语数据集生成合成的并行数据，然后用连续的反向翻译训练。最终模型扩展到了英语到许多方向，同时保持了多到英语的性能。我们称我们的方法为ecxtra（以英语为中心的跨语言（x）转移）。我们的方法依次利用辅助并行数据和单语言数据，并且在概念上很简单，仅在两个阶段都使用标准的跨熵目标。最终的ECXTRA模型对8种低资源语言的无监督NMT进行了评估，该语言为英语至哈萨克语（22.3> 10.4 bleu）以及其他15个翻译方向的竞争性能而获得了新的最先进。

Pre-training is an effective technique for ensuring robust performance on a variety of machine learning tasks. It typically depends on large-scale crawled corpora that can result in toxic or biased models. Such data can also be problematic with respect to copyright, attribution, and privacy. Pre-training with synthetic tasks and data is a promising way of alleviating such concerns since no real-world information is ingested by the model. Our goal in this paper is to understand what makes for a good pre-trained model when using synthetic resources. We answer this question in the context of neural machine translation by considering two novel approaches to translation model pre-training. Our first approach studies the effect of pre-training on obfuscated data derived from a parallel corpus by mapping words to a vocabulary of 'nonsense' tokens. Our second approach explores the effect of pre-training on procedurally generated synthetic parallel data that does not depend on any real human language corpus. Our empirical evaluation on multiple language pairs shows that, to a surprising degree, the benefits of pre-training can be realized even with obfuscated or purely synthetic parallel data. In our analysis, we consider the extent to which obfuscated and synthetic pre-training techniques can be used to mitigate the issue of hallucinated model toxicity.

在所有人类语言对之间实现通用翻译是机器翻译的圣杯（MT）研究。虽然最近在大量的多语言MT中的进展是达到这一目标的一步，但它变得明显，即简单地通过在更加平行数据上训练扩展多语言MT系统是不可编译的，因为用于低资源和非英语的标记数据的可用性 - 姓氏对禁止有限。为此，我们展示了一种务实的方法，可以使用监督和自我监督目标的混合来构建涵盖数百种语言的多语种MT模型，具体取决于不同语言对的数据可用性。我们展示这两种训练范例之间的协同作用使模型能够在零资源设置中产生高质量的翻译，甚至超过监控的用于中资和中资和中资质。我们开展广泛的实验，了解多语言监督，域错配和平行和单机数据量的效果，以了解我们自我监督的多语言模型的质量。为了展示方法的可扩展性，我们培训具有200多种语言的模型，并在几个先前研究的语言上展示了对零资源翻译的高性能。我们希望我们的调查结果将成为踏脚石，以便为下一千种语言进行翻译。

语言模型预训练的最新进展利用大规模数据集创建多语言模型。但是，这些数据集中大多遗漏了低资源语言。这主要是因为网络上没有很好地表示口语，因此被排除在用于创建数据集的大规模爬网中。此外，这些模型的下游用户仅限于最初选择用于预训练的语言的选择。这项工作调查了如何最佳利用现有的预培训模型来为16种非洲语言创建低资源翻译系统。我们关注两个问题：1）如何将预训练的模型用于初始预培训中未包含的语言？ 2）生成的翻译模型如何有效地转移到新域？为了回答这些问题，我们创建了一个新的非洲新闻语料库，涵盖16种语言，其中8种语言不属于任何现有评估数据集的一部分。我们证明，将两种语言转移到其他语言和其他领域的最有效策略是，以少量的高质量翻译数据微调大型预训练模型。

Multilingual machine translation models can benefit from synergy between different language pairs, but also suffer from interference. While there is a growing number of sophisticated methods that aim to eliminate interference, our understanding of interference as a phenomenon is still limited. This work identifies the main factors that contribute to interference in multilingual machine translation. Through systematic experimentation, we find that interference (or synergy) are primarily determined by model size, data size, and the proportion of each language pair within the total dataset. We observe that substantial interference occurs mainly when the model is very small with respect to the available training data, and that using standard transformer configurations with less than one billion parameters largely alleviates interference and promotes synergy. Moreover, we show that tuning the sampling temperature to control the proportion of each language pair in the data is key to balancing the amount of interference between low and high resource language pairs effectively, and can lead to superior performance overall.

Multilingual machine translation suffers from negative interference across languages. A common solution is to relax parameter sharing with language-specific modules like adapters. However, adapters of related languages are unable to transfer information, and their total number of parameters becomes prohibitively expensive as the number of languages grows. In this work, we overcome these drawbacks using hyper-adapters -- hyper-networks that generate adapters from language and layer embeddings. While past work had poor results when scaling hyper-networks, we propose a rescaling fix that significantly improves convergence and enables training larger hyper-networks. We find that hyper-adapters are more parameter efficient than regular adapters, reaching the same performance with up to 12 times less parameters. When using the same number of parameters and FLOPS, our approach consistently outperforms regular adapters. Also, hyper-adapters converge faster than alternative approaches and scale better than regular dense networks. Our analysis shows that hyper-adapters learn to encode language relatedness, enabling positive transfer across languages.

本文介绍了我们提交给WMT21共享新闻翻译任务的受限轨道。我们专注于三个相对低的资源语言对孟加拉，从印地语，英语往返Hausa，以及来自Zulu的Xhosa。为了克服相对低行数据的限制，我们使用采用并行和单晶体数据的多任务目标训练多语言模型。此外，我们使用后退转换增强数据。我们还培养了一种双语模型，包括后退转换和知识蒸馏，然后使用序列到序列映射来组合两种模型。我们看到迄今为止英语和来自Hausa的Bleu Point的相对收益约为70％，以及与双语基线相比，孟加拉和从Zulu的孟加拉和从Zulu的相对改善约25％。

以前的工作主要侧重于改善NLU任务的交叉传输，具有多语言预用编码器（MPE），或提高与伯特的监督机器翻译的性能。然而，探索了，MPE是否可以有助于促进NMT模型的交叉传递性。在本文中，我们专注于NMT中的零射频转移任务。在此任务中，NMT模型培训，只有一个语言对的并行数据集和搁置架MPE，然后它直接测试在零拍语言对上。我们为此任务提出了Sixt，一个简单而有效的模型。 SIXT利用了两阶段培训计划利用MPE，并进一步改进了解离编码器和容量增强的解码器。使用此方法，SIMPT显着优于MBart，这是一个用于NMT的预磨削的多语言编码器解码器模型，平均改善了14个源语言的零拍摄的任何英语测试集上的7.1 BLEU。此外，培训计算成本和培训数据较少，我们的模型在15个任何英语测试组上实现了比Criss和M2M-100，两个强大的多语言NMT基线更好的性能。

本报告介绍了在大型多语种计算机翻译中为WMT21共享任务的Microsoft的机器翻译系统。我们参加了所有三种评估轨道，包括大轨道和两个小轨道，前者是无约束的，后两者完全受约束。我们的模型提交到共享任务的初始化用deltalm \脚注{\ url {https://aka.ms/deltalm}}，一个通用的预训练的多语言编码器 - 解码器模型，并相应地使用巨大的收集并行进行微调数据和允许的数据源根据轨道设置，以及应用逐步学习和迭代背翻译方法进一步提高性能。我们的最终提交在自动评估度量方面排名第一的三条轨道。

在这项工作中，我们证明了多种语的大规模序列到序列（SEQ2SEQ）模型，该模型是通过Denoising和因果语言建模（CLM）任务的混合物进行训练的，比仅解码器模型更有效地进行了效率的学习者在各种任务上。特别是，我们培训了一个名为Alexa教师模型（Alexatm 20b）的200亿个参数多语言SEQ2SEQ模型，并表明它在1-Shot摘要任务上实现了最先进的（SOTA）性能，超过了更大的540B PALM DOPODER模型。 Alexatm 20b还可以在1-Shot Machine翻译中实现SOTA，尤其是对于低资源语言，几乎所有语言对（阿拉伯语，英语，法语，德语，德语，印地语，意大利语，日语，以及flores-101数据集上的泰卢固语）。我们还显示了零拍设置，AlexATM 20B在SuperGlue和SqueadV2数据集上的表现优于GPT3（175B），并在XNLI，XCOPA，PAWS-X和XWINOGRAD等多语言任务上提供SOTA性能。总体而言，我们的结果为SEQ2SEQ模型提供了一个令人信服的案例，作为大型语言模型（LLM）培训的仅解码器模型的强大替代方法。

神经机器翻译（NMT）模型在大型双语数据集上已有效。但是，现有的方法和技术表明，该模型的性能高度取决于培训数据中的示例数量。对于许多语言而言，拥有如此数量的语料库是一个牵强的梦想。我们从单语言词典探索新语言的单语扬声器中汲取灵感，我们研究了双语词典对具有极低或双语语料库的语言的适用性。在本文中，我们使用具有NMT模型的双语词典探索方法，以改善资源极低的资源语言的翻译。我们将此工作扩展到多语言系统，表现出零拍的属性。我们详细介绍了字典质量，培训数据集大小，语言家族等对翻译质量的影响。多种低资源测试语言的结果表明，我们的双语词典方法比基线相比。

最近在单语数据和机器翻译（MT）进行微调的预培训方面取得了成功，但尚不清楚如何最好地利用预先训练的模型来完成给定的MT任务。本文在微调MT上的预训练模型时研究了冻结参数的好处和缺点。我们专注于1）微调仅在英语单语言数据的BART上训练的模型。2）微调一个模型，该模型对25种语言的单语言数据进行了培训，Mbart。对于Bart，我们通过冻结大多数模型参数并添加额外的位置嵌入来获得最佳性能。对于MBART，我们将大多数语言对的天真微调的性能与编码器以及大多数解码器搭配。编码器的注意参数对于微调最重要。当将自己限制为越南人对英语的室外训练套装时，我们看到了基线的最大进步。

在完全共享所有语言参数的多语言神经机器翻译模型中，通常使用人工语言令牌来指导转换为所需的目标语言。但是，最近的研究表明，预备语言代币有时无法将多语言神经机器翻译模型导航到正确的翻译方向，尤其是在零弹性翻译上。为了减轻此问题，我们提出了两种方法：语言嵌入实施例和语言意识的多头关注，以学习信息丰富的语言表示，以将翻译转换为正确的方向。前者体现了沿着从源到目标的信息流中的不同关键切换点的语言，旨在放大翻译方向引导信号。后者利用矩阵而不是向量来表示连续空间中的语言。矩阵分为多个头，以学习多个子空间中的语言表示。在两个数据集上进行大规模多语言神经机器翻译的实验结果表明，语言意识到的多头注意力受益于监督和零弹性翻译，并大大减轻了脱靶翻译问题。进一步的语言类型学预测实验表明，通过我们的方法学到的基于基质的语言表示能够捕获丰富的语言类型学特征。

Traditional multilingual neural machine translation (MNMT) uses a single model to translate all directions. However, with the increasing scale of language pairs, simply using a single model for massive MNMT brings new challenges: parameter tension and large computations. In this paper, we revisit multi-way structures by assigning an individual branch for each language (group). Despite being a simple architecture, it is challenging to train de-centralized models due to the lack of constraints to align representations from all languages. We propose a localized training recipe to map different branches into a unified space, resulting in an efficient detachable model, Lego-MT. For a fair comparison, we collect data from OPUS and build the first large-scale open-source translation benchmark covering 7 language-centric data, each containing 445 language pairs. Experiments show that Lego-MT (1.2B) brings gains of more than 4 BLEU while outperforming M2M-100 (12B) (We will public all training data, models, and checkpoints)

This paper demonstrates that multilingual denoising pre-training produces significant performance gains across a wide variety of machine translation (MT) tasks. We present mBART -a sequence-to-sequence denoising auto-encoder pre-trained on large-scale monolingual corpora in many languages using the BART objective . mBART is the first method for pre-training a complete sequence-to-sequence model by denoising full texts in multiple languages, while previous approaches have focused only on the encoder, decoder, or reconstructing parts of the text. Pre-training a complete model allows it to be directly fine tuned for supervised (both sentence-level and document-level) and unsupervised machine translation, with no task-specific modifications. We demonstrate that adding mBART initialization produces performance gains in all but the highest-resource settings, including up to 12 BLEU points for low resource MT and over 5 BLEU points for many document-level and unsupervised models. We also show it also enables new types of transfer to language pairs with no bi-text or that were not in the pre-training corpus, and present extensive analysis of which factors contribute the most to effective pre-training.

We present SpeechMatrix, a large-scale multilingual corpus of speech-to-speech translations mined from real speech of European Parliament recordings. It contains speech alignments in 136 language pairs with a total of 418 thousand hours of speech. To evaluate the quality of this parallel speech, we train bilingual speech-to-speech translation models on mined data only and establish extensive baseline results on EuroParl-ST, VoxPopuli and FLEURS test sets. Enabled by the multilinguality of SpeechMatrix, we also explore multilingual speech-to-speech translation, a topic which was addressed by few other works. We also demonstrate that model pre-training and sparse scaling using Mixture-of-Experts bring large gains to translation performance. The mined data and models are freely available.