Compared to conventional bilingual translation systems, massively multilingual machine translation is appealing because a single model can translate into multiple languages and benefit from knowledge transfer for low resource languages. On the other hand, massively multilingual models suffer from the curse of multilinguality, unless scaling their size massively, which increases their training and inference costs. Sparse Mixture-of-Experts models are a way to drastically increase model capacity without the need for a proportional amount of computing. The recently released NLLB-200 is an example of such a model. It covers 202 languages but requires at least four 32GB GPUs just for inference. In this work, we propose a pruning method that allows the removal of up to 80\% of experts with a negligible loss in translation quality, which makes it feasible to run the model on a single 32GB GPU. Further analysis suggests that our pruning metrics allow to identify language-specific experts and prune non-relevant experts for a given language pair.

Sparsely gated Mixture of Experts (MoE) models have been shown to be a compute-efficient method to scale model capacity for multilingual machine translation. However, for low-resource tasks, MoE models severely over-fit. We show effective regularization strategies, namely dropout techniques for MoE layers in EOM and FOM, Conditional MoE Routing and Curriculum Learning methods that prevent over-fitting and improve the performance of MoE models on low-resource tasks without adversely affecting high-resource tasks. On a massively multilingual machine translation benchmark, our strategies result in about +1 chrF++ improvement in very low resource language pairs. We perform an extensive analysis of the learned MoE routing to better understand the impact of our regularization methods and how we can improve them.

多语种NMT已成为MT在生产中部署的有吸引力的解决方案。但是要匹配双语质量，它符合较大且较慢的型号。在这项工作中，我们考虑了几种方法在推理时更快地使多语言NMT变得更快而不会降低其质量。我们在两种20语言多平行设置中尝试几个“光解码器”架构：在TED会谈中小规模和帕拉克曲线上的大规模。我们的实验表明，将具有词汇过滤的浅解码器组合在于，在翻译质量下没有损失的速度超过两倍。我们用Bleu和Chrf（380语言对），鲁棒性评估和人类评估验证了我们的研究结果。

稀疏激活的变压器（例如专家的混合物（MOE））由于其极端的缩放能力而引起了极大的兴趣，这可以使模型大小的急剧增加而没有大幅增加计算成本。为了实现这一目标，MOE模型用变压器中的Experts子层取代了前馈子层，并使用门控网络将每个令牌路由到其指定的专家。由于对此类模型进行有效培训的共同实践需要在不同的机器上分发专家和代币，因此这种路由策略通常会产生巨大的跨机器通信成本，因为代币及其分配的专家可能居住在不同的机器中。在本文中，我们提出了\ emph {门控辍学}，它允许代币忽略门控网络并留在其本地机器，从而减少了交叉机器的通信。与传统辍学类似，我们还表明，门控辍学在训练过程中具有正规化效果，从而改善了概括性能。我们验证了对多语言机器翻译任务中门控辍学的有效性。我们的结果表明，门控辍学可改善具有更快的壁式时间收敛速率的最先进的MOE模型，并为各种模型尺寸和数据集提供更好的BLEU分数。

Multi-head self-attention is a key component of the Transformer, a state-of-the-art architecture for neural machine translation. In this work we evaluate the contribution made by individual attention heads in the encoder to the overall performance of the model and analyze the roles played by them. We find that the most important and confident heads play consistent and often linguistically-interpretable roles. When pruning heads using a method based on stochastic gates and a differentiable relaxation of the L 0 penalty, we observe that specialized heads are last to be pruned. Our novel pruning method removes the vast majority of heads without seriously affecting performance. For example, on the English-Russian WMT dataset, pruning 38 out of 48 encoder heads results in a drop of only 0.15 BLEU. 1

Multilingual machine translation models can benefit from synergy between different language pairs, but also suffer from interference. While there is a growing number of sophisticated methods that aim to eliminate interference, our understanding of interference as a phenomenon is still limited. This work identifies the main factors that contribute to interference in multilingual machine translation. Through systematic experimentation, we find that interference (or synergy) are primarily determined by model size, data size, and the proportion of each language pair within the total dataset. We observe that substantial interference occurs mainly when the model is very small with respect to the available training data, and that using standard transformer configurations with less than one billion parameters largely alleviates interference and promotes synergy. Moreover, we show that tuning the sampling temperature to control the proportion of each language pair in the data is key to balancing the amount of interference between low and high resource language pairs effectively, and can lead to superior performance overall.

We present SpeechMatrix, a large-scale multilingual corpus of speech-to-speech translations mined from real speech of European Parliament recordings. It contains speech alignments in 136 language pairs with a total of 418 thousand hours of speech. To evaluate the quality of this parallel speech, we train bilingual speech-to-speech translation models on mined data only and establish extensive baseline results on EuroParl-ST, VoxPopuli and FLEURS test sets. Enabled by the multilinguality of SpeechMatrix, we also explore multilingual speech-to-speech translation, a topic which was addressed by few other works. We also demonstrate that model pre-training and sparse scaling using Mixture-of-Experts bring large gains to translation performance. The mined data and models are freely available.

Recently, very large pre-trained models achieve state-of-the-art results in various natural language processing (NLP) tasks, but their size makes it more challenging to apply them in resource-constrained environments. Compression techniques allow to drastically reduce the size of the models and therefore their inference time with negligible impact on top-tier metrics. However, the general performance averaged across multiple tasks and/or languages may hide a drastic performance drop on under-represented features, which could result in the amplification of biases encoded by the models. In this work, we assess the impact of compression methods on Multilingual Neural Machine Translation models (MNMT) for various language groups, gender, and semantic biases by extensive analysis of compressed models on different machine translation benchmarks, i.e. FLORES-101, MT-Gender, and DiBiMT. We show that the performance of under-represented languages drops significantly, while the average BLEU metric only slightly decreases. Interestingly, the removal of noisy memorization with compression leads to a significant improvement for some medium-resource languages. Finally, we demonstrate that compression amplifies intrinsic gender and semantic biases, even in high-resource languages. Code: https://github.com/alirezamshi/bias-compressedMT

稀疏的专家模型是一个三十年来的概念，作为深度学习中流行的建筑。这类体系结构包括专家的混合物，交换变压器，路由网络，基础层等，所有这些都以一个统一的想法，即每个示例都由参数的一个子集进行。通过这样做，稀疏度将参数计数与每个示例的计算分解，从而允许使用极大但有效的模型。最终的模型显示了各种领域的显着改善，例如自然语言处理，计算机视觉和语音识别。我们回顾了稀疏专家模型的概念，提供了对常见算法的基本描述，将深度学习时代的进步进行上下文化，并通过突出未来工作的领域来结束。

Attention is a powerful and ubiquitous mechanism for allowing neural models to focus on particular salient pieces of information by taking their weighted average when making predictions. In particular, multi-headed attention is a driving force behind many recent state-of-the-art natural language processing (NLP) models such as Transformer-based MT models and BERT. These models apply multiple attention mechanisms in parallel, with each attention "head" potentially focusing on different parts of the input, which makes it possible to express sophisticated functions beyond the simple weighted average. In this paper we make the surprising observation that even if models have been trained using multiple heads, in practice, a large percentage of attention heads can be removed at test time without significantly impacting performance. In fact, some layers can even be reduced to a single head. We further examine greedy algorithms for pruning down models, and the potential speed, memory efficiency, and accuracy improvements obtainable therefrom. Finally, we analyze the results with respect to which parts of the model are more reliant on having multiple heads, and provide precursory evidence that training dynamics play a role in the gains provided by multi-head attention 1 .1 Code to replicate our experiments is provided at https://github.com/pmichel31415/ are-16-heads-really-better-than-1

在深度学习中，模型通常重用所有输入的相同参数。专家的混合（MOE）违反了这一点，而是为每个传入示例选择不同的参数。结果是一个稀疏激活的模型 - 具有残酷数量的参数 - 但恒定的计算成本。然而，尽管MOE取得了一些显着的成功，但复杂性，沟通成本和培训不稳定的阻碍了广泛的采用 - 我们使用Switch Transformer解决了这些领域。我们简化了MOE路由算法和设计直观的改进模型，以降低的通信和计算成本。我们提出的培训技术有助于纠缠不稳定，我们表明稀疏模型可能首次以较低的精度（BFLOAT16）格式进行培训。我们设计了基于T5基数和T5总数的模型，以使用相同的计算资源获得高达7倍的训练速度。这些改进扩展到多语言设置，我们在所有101种语言中衡量对MT5基本版本的收益。最后，我们通过在“巨大的清洁爬行语料库”上预先培训高达数万亿个参数模型，并在T5-XXL模型上实现4倍的速度，从而提高了语言模型的当前规模。

多语种神经机翻译（MNMT）旨在通过单一模型进行翻译多种语言，并且由于具有共享参数的不同语言的有效知识传输，已被证明是成功的。但是，它仍然是一个开放的问题，应该共享哪些参数，并且需要是特定于任务的。目前，常识是启发式设计或搜索特定语言的模块，这很难找到最佳配置。在本文中，我们提出了一种基于新的参数差异化方法，允许模型确定在训练期间应该是哪个参数。灵感来自蜂窝分化，我们方法中的每个共享参数都可以动态区分为更专业化的类型。我们还将差分标准定义为任务间梯度相似性。因此，突出的任务渐变间的参数更可能是特定于语言的。关于多语言数据集的大量实验表明，我们的方法显着优于不同参数共享配置的各种强基线。进一步的分析表明，通过我们的方法获得的参数共享配置与语言近似度很好地相关。

Multilingual machine translation suffers from negative interference across languages. A common solution is to relax parameter sharing with language-specific modules like adapters. However, adapters of related languages are unable to transfer information, and their total number of parameters becomes prohibitively expensive as the number of languages grows. In this work, we overcome these drawbacks using hyper-adapters -- hyper-networks that generate adapters from language and layer embeddings. While past work had poor results when scaling hyper-networks, we propose a rescaling fix that significantly improves convergence and enables training larger hyper-networks. We find that hyper-adapters are more parameter efficient than regular adapters, reaching the same performance with up to 12 times less parameters. When using the same number of parameters and FLOPS, our approach consistently outperforms regular adapters. Also, hyper-adapters converge faster than alternative approaches and scale better than regular dense networks. Our analysis shows that hyper-adapters learn to encode language relatedness, enabling positive transfer across languages.

只有在模型在大规模的多语言环境中培训的情况下，才有可能在无监督的机器翻译（UMT）上进行无监督的机器翻译（UMT），这意味着有能力的无监督翻译（例如尼泊尔或辛哈拉）的胜任的不受监督的翻译，例如尼泊尔或辛哈拉语。与高资源对应物混合。尽管如此，尽管高资源语言极大地帮助启动了目标低资源翻译任务，但它们之间的语言差异可能会阻碍他们的进一步改进。在这项工作中，我们提出了一个简单的完善程序，以将语言与预先训练的多语言UMT模型相关联，以仅关注目标低资源任务。我们的方法在完全无监督的翻译任务中实现了最新的尼泊尔，僧伽罗，古吉拉特语，拉脱维亚，爱沙尼亚和哈萨克的最新技术，分别为3.5、3.3、3.3、4.1、4.2、4.2和3.3。我们的代码库可从https://github.com/nxphi47/refine_unsup_multlingual_mt获得

We propose a simple solution to use a single Neural Machine Translation (NMT) model to translate between multiple languages. Our solution requires no changes to the model architecture from a standard NMT system but instead introduces an artificial token at the beginning of the input sentence to specify the required target language. The rest of the model, which includes an encoder, decoder and attention module, remains unchanged and is shared across all languages. Using a shared wordpiece vocabulary, our approach enables Multilingual NMT using a single model without any increase in parameters, which is significantly simpler than previous proposals for Multilingual NMT. On the WMT'14 benchmarks, a single multilingual model achieves comparable performance for English→French and surpasses state-of-the-art results for English→German. Similarly, a single multilingual model surpasses state-of-the-art results for French→English and German→English on WMT'14 and WMT'15 benchmarks, respectively. On production corpora, multilingual models of up to twelve language pairs allow for better translation of many individual pairs. In addition to improving the translation quality of language pairs that the model was trained with, our models can also learn to perform implicit bridging between language pairs never seen explicitly during training, showing that transfer learning and zero-shot translation is possible for neural translation. Finally, we show analyses that hints at a universal interlingua representation in our models and show some interesting examples when mixing languages.

专家（MOE）的稀疏混合物由于具有负担得起的计算开销而有希望的缩放能力，因此引起了极大的兴趣。 Moe将密集的层转换为稀疏的专家，并利用封闭式路由网络使专家有条件地激活。但是，随着专家的数量的增长，带有残酷参数的MOE会受到过度拟合和稀疏数据分配的影响。此类问题在数据有限的任务上尤为严重，因此阻碍了MOE模型通过扩展来提高性能的进度。在这项工作中，我们提出了专家群集的混合 - 一种通用方法，可以使专家层通过在路由阶段施加基于方差的约束来学习更多多样化和适当的知识。我们进一步提出了专门为专家集群结构设计的集群级专家辍学策略。我们的实验表明，MEEC可以提高机器翻译和自然语言理解任务的性能，并提高在有限数据下扩展专家的性能上限。我们还验证了MEEC在缓解过度拟合和稀疏数据分配中起积极的作用。

最近非自动增加（NAR）机器翻译最近取得了显着的改进，现在优于一些基准测试的自动增加（AR）模型，为AR推断提供有效的替代方案。然而，虽然AR转换通常使用多语言模型来实现，但是从语言之间的转移和改善的服务效率，多语言NAR模型仍然相对未开发。作为一个示例NAR模型和变压器作为半NAR模型，采用连接员时间分类（CTC），我们展示了多语种NAR的全面实证研究。我们在容量限制下对相关语言与负转移之间的积极转移来测试其能力。随着NAR模型需要蒸馏培训套，我们仔细研究双语与多语种教师的影响。最后，我们适合多语言NAR的缩放法，这使得其相对于AR模型的性能随着模型量表的增加而定量。

最近在单语数据和机器翻译（MT）进行微调的预培训方面取得了成功，但尚不清楚如何最好地利用预先训练的模型来完成给定的MT任务。本文在微调MT上的预训练模型时研究了冻结参数的好处和缺点。我们专注于1）微调仅在英语单语言数据的BART上训练的模型。2）微调一个模型，该模型对25种语言的单语言数据进行了培训，Mbart。对于Bart，我们通过冻结大多数模型参数并添加额外的位置嵌入来获得最佳性能。对于MBART，我们将大多数语言对的天真微调的性能与编码器以及大多数解码器搭配。编码器的注意参数对于微调最重要。当将自己限制为越南人对英语的室外训练套装时，我们看到了基线的最大进步。

The word alignment task, despite its prominence in the era of statistical machine translation (SMT), is niche and under-explored today. In this two-part tutorial, we argue for the continued relevance for word alignment. The first part provides a historical background to word alignment as a core component of the traditional SMT pipeline. We zero-in on GIZA++, an unsupervised, statistical word aligner with surprising longevity. Jumping forward to the era of neural machine translation (NMT), we show how insights from word alignment inspired the attention mechanism fundamental to present-day NMT. The second part shifts to a survey approach. We cover neural word aligners, showing the slow but steady progress towards surpassing GIZA++ performance. Finally, we cover the present-day applications of word alignment, from cross-lingual annotation projection, to improving translation.

We introduce EdgeFormer -- a parameter-efficient Transformer for on-device seq2seq generation under the strict computation and memory constraints. Compared with the previous parameter-efficient Transformers, EdgeFormer applies two novel principles for cost-effective parameterization, allowing it to perform better given the same parameter budget; moreover, EdgeFormer is further enhanced by layer adaptation innovation that is proposed for improving the network with shared layers. Extensive experiments show EdgeFormer can effectively outperform previous parameter-efficient Transformer baselines and achieve competitive results under both the computation and memory constraints. Given the promising results, we release EdgeLM -- the pretrained version of EdgeFormer, which is the first publicly available pretrained on-device seq2seq model that can be easily fine-tuned for seq2seq tasks with strong results, facilitating on-device seq2seq generation in practice.