We present SpeechMatrix, a large-scale multilingual corpus of speech-to-speech translations mined from real speech of European Parliament recordings. It contains speech alignments in 136 language pairs with a total of 418 thousand hours of speech. To evaluate the quality of this parallel speech, we train bilingual speech-to-speech translation models on mined data only and establish extensive baseline results on EuroParl-ST, VoxPopuli and FLEURS test sets. Enabled by the multilinguality of SpeechMatrix, we also explore multilingual speech-to-speech translation, a topic which was addressed by few other works. We also demonstrate that model pre-training and sparse scaling using Mixture-of-Experts bring large gains to translation performance. The mined data and models are freely available.

我们介绍了一种无线文字语音转换（S2ST）系统，可以将来自一种语言的语音转换为另一种语言，并且可以在不需要任何文本数据的情况下构建。与文献中的现有工作不同，我们解决了模拟多扬声器目标语音的挑战，并用现实世界的S2ST数据训练系统。我们方法的关键是一种自我监督的单位语音标准化技术，该标准化技术将预先训练的语音编码器具有来自多个扬声器的配对声音，以及单个参考扬声器，以减少由于复印件引起的变化，同时保留词汇内容。只有10分钟的语音标准化的配对数据，我们在培训\ vp〜s2st数据集上的S2ST模型时获得平均3.2 BLEU增益，而不是在未标准化的语音目标上培训的基线。我们还将自动开采的S2ST数据纳入并显示额外的2.0 BLEU增益。据我们所知，我们是第一个建立无线的S2ST技术，可以用真实世界的数据培训，并为多种语言配对工作。

We present a new approach to perform zero-shot cross-modal transfer between speech and text for translation tasks. Multilingual speech and text are encoded in a joint fixed-size representation space. Then, we compare different approaches to decode these multimodal and multilingual fixed-size representations, enabling zero-shot translation between languages and modalities. All our models are trained without the need of cross-modal labeled translation data. Despite a fixed-size representation, we achieve very competitive results on several text and speech translation tasks. In particular, we significantly improve the state-of-the-art for zero-shot speech translation on Must-C. Incorporating a speech decoder in our framework, we introduce the first results for zero-shot direct speech-to-speech and text-to-speech translation.

本文介绍了基于Wav2VEC 2.0的跨语言语音表示学习的大规模模型。我们在128种语言中培训最多2B个公共讲话音频的近半小时的型号的模型，比公共数据的数量级比最大的已知事先工作。我们的评估涵盖了广泛的任务，域，数据制度和语言，都是高低资源。在Covost-2语音翻译基准测试中，我们将先前的最先进的状态平均为7.4 BLEU超过21个翻译方向进入英语。对于语音识别，XLS-R在Babel，MLS，CommonVoice以及Voxpopuli上的最佳已知工作中提高，降低了相对的误差率14-34％。 XLS-R还在Voxlingua107语言识别上设置了新的技术状态。此外，我们表明，具有足够的模型规模，交叉思维预先预测可以在将英语演讲翻译成其他语言时才能优于英语撇印，这是一个有利于单晶的预借预制的设置。我们希望XLS-R可以帮助改善世界上更多语言的语音处理任务。

Direct speech-to-speech translation (S2ST), in which all components can be optimized jointly, is advantageous over cascaded approaches to achieve fast inference with a simplified pipeline. We present a novel two-pass direct S2ST architecture, {\textit UnitY}, which first generates textual representations and predicts discrete acoustic units subsequently. We enhance the model performance by subword prediction in the first-pass decoder, advanced two-pass decoder architecture design and search strategy, and better training regularization. To leverage large amounts of unlabeled text data, we pre-train the first-pass text decoder based on the self-supervised denoising auto-encoding task. Experimental evaluations on benchmark datasets at various data scales demonstrate that UnitY outperforms a single-pass speech-to-unit translation model by 2.5-4.2 ASR-BLEU with 2.83x decoding speed-up. We show that the proposed methods boost the performance even when predicting spectrogram in the second pass. However, predicting discrete units achieves 2.51x decoding speed-up compared to that case.

我们介绍了CVSS，这是一种大规模的多语言对语音转换（S2ST）语料库，从21种语言覆盖了21种语言的句子级并行S2ST对。通过将Covost 2从Covost 2的翻译文本综合将翻译文本与最先进的TTS系统合成语音，源自公共语音语音语料库和COVOST 2语音到文本转换（ST）语料库。提供了两个版本的翻译演讲：1）CVSS-C：所有翻译演讲都是一种高质量的规范声音; 2）CVSS-T：翻译语音从相应的源语音传输。此外，CVSS提供标准化的翻译文本，它与翻译语音中的发音匹配。在每个版本的CVSS上，我们建立了基线多语言直接S2ST模型和Cascade S2ST模型，验证了语料库的有效性。为了构建强大的Cascade S2ST基准，我们在Covost 2上培训了St模型，这优于前一种最先进的培训，而无需额外的数据。尽管如此，直接S2ST模型的性能在从头开始训练时接近强级联基线，并且在匹配ST模型中初始化时，仅在ASR转换转换时的0.1或0.7bleu差异。

End-to-End speech-to-speech translation (S2ST) is generally evaluated with text-based metrics. This means that generated speech has to be automatically transcribed, making the evaluation dependent on the availability and quality of automatic speech recognition (ASR) systems. In this paper, we propose a text-free evaluation metric for end-to-end S2ST, named BLASER, to avoid the dependency on ASR systems. BLASER leverages a multilingual multimodal encoder to directly encode the speech segments for source input, translation output and reference into a shared embedding space and computes a score of the translation quality that can be used as a proxy to human evaluation. To evaluate our approach, we construct training and evaluation sets from more than 40k human annotations covering seven language directions. The best results of BLASER are achieved by training with supervision from human rating scores. We show that when evaluated at the sentence level, BLASER correlates significantly better with human judgment compared to ASR-dependent metrics including ASR-SENTBLEU in all translation directions and ASR-COMET in five of them. Our analysis shows combining speech and text as inputs to BLASER does not increase the correlation with human scores, but best correlations are achieved when using speech, which motivates the goal of our research. Moreover, we show that using ASR for references is detrimental for text-based metrics.

我们呈现TranslatOrron 2，一个神经直接语音转换转换模型，可以训练结束到底。 TranslatOrron 2由语音编码器，音素解码器，MEL谱图合成器和连接所有前三个组件的注意模块组成。实验结果表明，翻译ron 2在翻译质量和预测的语音自然方面，通过大幅度优于原始翻译，并且通过减轻超越，例如唠叨或长暂停来大幅提高预测演讲的鲁棒性。我们还提出了一种在翻译语音中保留源代言人声音的新方法。训练有素的模型被限制为保留源扬声器的声音，但与原始翻译ron不同，它无法以不同的扬声器的语音产生语音，使模型对生产部署更加强大，通过减轻潜在的滥用来创建欺骗音频伪影。当新方法与基于简单的替代的数据增强一起使用时，训练的翻译器2模型能够保留每个扬声器的声音，以便用扬声器转动输入输入。

直接语音到语音翻译（S2ST）模型与传统级联系统可用的数据量相比，几乎没有平行的S2ST数据遇到数据稀缺问题，该数据包括自动语音识别（ASR），机器翻译（MT）和文本到语音（TTS）合成。在这项工作中，我们使用未标记的语音数据和数据扩展来探索自我监督的预训练，以解决此问题。我们利用了最近提出的语音到单位翻译（S2UT）框架，该框架将目标语音编码为离散表示形式，并转移前训练前和有效的部分填充技术，可很好地适用于语音到文本翻译（S2T）通过研究语音编码器和离散单位解码器预训练，S2UT域。我们在西班牙语 - 英语翻译上进行的实验表明，与多任务学习相比，自我监督的预训练始终如一地提高模型性能，平均为6.6-12.1 BLEU增长，并且可以与数据增强技术相结合，以应用MT来创建弱监督监督的培训数据。音频样本可在以下网址获得：https：//facebookresearch.github.io/speech_translation/enhanced_direct_s2st_units/index.html。

We study the capabilities of speech processing systems trained simply to predict large amounts of transcripts of audio on the internet. When scaled to 680,000 hours of multilingual and multitask supervision, the resulting models generalize well to standard benchmarks and are often competitive with prior fully supervised results but in a zero-shot transfer setting without the need for any fine-tuning. When compared to humans, the models approach their accuracy and robustness. We are releasing models and inference code to serve as a foundation for further work on robust speech processing.

Data scarcity is one of the main issues with the end-to-end approach for Speech Translation, as compared to the cascaded one. Although most data resources for Speech Translation are originally document-level, they offer a sentence-level view, which can be directly used during training. But this sentence-level view is single and static, potentially limiting the utility of the data. Our proposed data augmentation method SegAugment challenges this idea and aims to increase data availability by providing multiple alternative sentence-level views of a dataset. Our method heavily relies on an Audio Segmentation system to re-segment the speech of each document, after which we obtain the target text with alignment methods. The Audio Segmentation system can be parameterized with different length constraints, thus giving us access to multiple and diverse sentence-level views for each document. Experiments in MuST-C show consistent gains across 8 language pairs, with an average increase of 2.2 BLEU points, and up to 4.7 BLEU for lower-resource scenarios in mTEDx. Additionally, we find that SegAugment is also applicable to purely sentence-level data, as in CoVoST, and that it enables Speech Translation models to completely close the gap between the gold and automatic segmentation at inference time.

语音翻译模型无法直接处理较长的音频，例如TED Talks，必须将其分为较短的段。语音翻译数据集提供了音频的手动分割，这些音频在现实世界中不可用，而现有的分割方法通常会在推理时大大降低翻译质量。为了弥合训练的手动分割与推理的自动分割之间的差距，我们提出了有监督的混合音频分割（SHAS），该方法可以有效地从任何手动分段语音语料库中学习最佳分割。首先，我们使用预先训练的WAV2VEC 2.0的语音表示形式来训练分类器，以识别分段中所包含的帧。然后，通过概率分裂和诱导算法找到最佳的分裂点，该算法逐渐在最低概率的框架下逐渐分裂，直到所有段都低于预先指定的长度为止。在Mast-C和MedX上进行的实验表明，通过我们的方法生成的片段的翻译方法将手动分割的质量在5个语言对上进行质量。也就是说，SHAS保留了手动细分的95-98％的BLEU分数，而现有方法的87-93％。我们的方法还可以推广到不同的域，并以看不见的语言实现高零弹性性能。

Speech translation (ST) is the task of directly translating acoustic speech signals in a source language into text in a foreign language. ST task has been addressed, for a long time, using a pipeline approach with two modules : first an Automatic Speech Recognition (ASR) in the source language followed by a text-to-text Machine translation (MT). In the past few years, we have seen a paradigm shift towards the end-to-end approaches using sequence-to-sequence deep neural network models. This paper presents our efforts towards the development of the first Broadcast News end-to-end Arabic to English speech translation system. Starting from independent ASR and MT LDC releases, we were able to identify about 92 hours of Arabic audio recordings for which the manual transcription was also translated into English at the segment level. These data was used to train and compare pipeline and end-to-end speech translation systems under multiple scenarios including transfer learning and data augmentation techniques.

本文介绍了我们针对IWSLT 2022离线任务的端到端Yitrans语音翻译系统的提交，该任务从英语音频转换为德语，中文和日语。 Yitrans系统建立在大规模训练的编码器模型上。更具体地说，我们首先设计了多阶段的预训练策略，以建立具有大量标记和未标记数据的多模式模型。然后，我们为下游语音翻译任务微调模型的相应组件。此外，我们做出了各种努力，以提高性能，例如数据过滤，数据增强，语音细分，模型集合等。实验结果表明，我们的Yitrans系统比在三个翻译方向上的强基线取得了显着改进，并且比去年在TST2021英语 - 德国人中的最佳端到端系统方面的改进+5.2 BLEU改进。根据自动评估指标，我们的最终意见在英语 - 德国和英语端到端系统上排名第一。我们使代码和模型公开可用。

We propose a simple solution to use a single Neural Machine Translation (NMT) model to translate between multiple languages. Our solution requires no changes to the model architecture from a standard NMT system but instead introduces an artificial token at the beginning of the input sentence to specify the required target language. The rest of the model, which includes an encoder, decoder and attention module, remains unchanged and is shared across all languages. Using a shared wordpiece vocabulary, our approach enables Multilingual NMT using a single model without any increase in parameters, which is significantly simpler than previous proposals for Multilingual NMT. On the WMT'14 benchmarks, a single multilingual model achieves comparable performance for English→French and surpasses state-of-the-art results for English→German. Similarly, a single multilingual model surpasses state-of-the-art results for French→English and German→English on WMT'14 and WMT'15 benchmarks, respectively. On production corpora, multilingual models of up to twelve language pairs allow for better translation of many individual pairs. In addition to improving the translation quality of language pairs that the model was trained with, our models can also learn to perform implicit bridging between language pairs never seen explicitly during training, showing that transfer learning and zero-shot translation is possible for neural translation. Finally, we show analyses that hints at a universal interlingua representation in our models and show some interesting examples when mixing languages.

字幕（替代）的语音翻译是通过将符合特定显示指南的字幕划分插入字幕中断，将语音数据自动转化为良好的字幕。与语音翻译（ST）类似，模型训练需要并行数据，其中包括音频输入与其文本翻译配对。然而，在替代方面，还必须用字幕断裂来注释文本。到目前为止，这一要求代表了系统开发的瓶颈，如公开可用的替代公司所证实。为了填补这一空白，我们提出了一种在不干预的情况下将现有的ST Corpora转换为替代资源的方法。我们构建了一个分段模型，该模型通过以多模式的方式利用音频和文本来自动将文本片段分为适当的字幕，从而在零拍摄条件下实现了高分子的质量。对手动和自动分割培训的替代系统的比较实验导致相似的性能，显示了我们方法的有效性。

This paper demonstrates that multilingual denoising pre-training produces significant performance gains across a wide variety of machine translation (MT) tasks. We present mBART -a sequence-to-sequence denoising auto-encoder pre-trained on large-scale monolingual corpora in many languages using the BART objective . mBART is the first method for pre-training a complete sequence-to-sequence model by denoising full texts in multiple languages, while previous approaches have focused only on the encoder, decoder, or reconstructing parts of the text. Pre-training a complete model allows it to be directly fine tuned for supervised (both sentence-level and document-level) and unsupervised machine translation, with no task-specific modifications. We demonstrate that adding mBART initialization produces performance gains in all but the highest-resource settings, including up to 12 BLEU points for low resource MT and over 5 BLEU points for many document-level and unsupervised models. We also show it also enables new types of transfer to language pairs with no bi-text or that were not in the pre-training corpus, and present extensive analysis of which factors contribute the most to effective pre-training.

本报告介绍了在大型多语种计算机翻译中为WMT21共享任务的Microsoft的机器翻译系统。我们参加了所有三种评估轨道，包括大轨道和两个小轨道，前者是无约束的，后两者完全受约束。我们的模型提交到共享任务的初始化用deltalm \脚注{\ url {https://aka.ms/deltalm}}，一个通用的预训练的多语言编码器 - 解码器模型，并相应地使用巨大的收集并行进行微调数据和允许的数据源根据轨道设置，以及应用逐步学习和迭代背翻译方法进一步提高性能。我们的最终提交在自动评估度量方面排名第一的三条轨道。

在本文中，我们介绍了一个高质量的大规模基准数据集，用于英语 - 越南语音翻译，其中有508音频小时，由331k的三胞胎组成（句子长度的音频，英语源笔录句，越南人目标subtitle句子）。我们还使用强基础进行了经验实验，发现传统的“级联”方法仍然优于现代“端到端”方法。据我们所知，这是第一个大规模的英语 - 越南语音翻译研究。我们希望我们的公开数据集和研究都可以作为未来研究和英语语音翻译应用的起点。我们的数据集可从https://github.com/vinairesearch/phost获得

End-to-end Speech Translation (E2E ST) aims to translate source speech into target translation without generating the intermediate transcript. However, existing approaches for E2E ST degrade considerably when only limited ST data are available. We observe that an ST model's performance strongly correlates with its embedding similarity from speech and transcript. In this paper, we propose Word-Aligned COntrastive learning (WACO), a novel method for few-shot speech-to-text translation. Our key idea is bridging word-level representations for both modalities via contrastive learning. We evaluate WACO and other methods on the MuST-C dataset, a widely used ST benchmark. Our experiments demonstrate that WACO outperforms the best baseline methods by 0.7-8.5 BLEU points with only 1-hour parallel data. Code is available at https://anonymous.4open.science/r/WACO .