语音翻译模型无法直接处理较长的音频，例如TED Talks，必须将其分为较短的段。语音翻译数据集提供了音频的手动分割，这些音频在现实世界中不可用，而现有的分割方法通常会在推理时大大降低翻译质量。为了弥合训练的手动分割与推理的自动分割之间的差距，我们提出了有监督的混合音频分割（SHAS），该方法可以有效地从任何手动分段语音语料库中学习最佳分割。首先，我们使用预先训练的WAV2VEC 2.0的语音表示形式来训练分类器，以识别分段中所包含的帧。然后，通过概率分裂和诱导算法找到最佳的分裂点，该算法逐渐在最低概率的框架下逐渐分裂，直到所有段都低于预先指定的长度为止。在Mast-C和MedX上进行的实验表明，通过我们的方法生成的片段的翻译方法将手动分割的质量在5个语言对上进行质量。也就是说，SHAS保留了手动细分的95-98％的BLEU分数，而现有方法的87-93％。我们的方法还可以推广到不同的域，并以看不见的语言实现高零弹性性能。

Data scarcity is one of the main issues with the end-to-end approach for Speech Translation, as compared to the cascaded one. Although most data resources for Speech Translation are originally document-level, they offer a sentence-level view, which can be directly used during training. But this sentence-level view is single and static, potentially limiting the utility of the data. Our proposed data augmentation method SegAugment challenges this idea and aims to increase data availability by providing multiple alternative sentence-level views of a dataset. Our method heavily relies on an Audio Segmentation system to re-segment the speech of each document, after which we obtain the target text with alignment methods. The Audio Segmentation system can be parameterized with different length constraints, thus giving us access to multiple and diverse sentence-level views for each document. Experiments in MuST-C show consistent gains across 8 language pairs, with an average increase of 2.2 BLEU points, and up to 4.7 BLEU for lower-resource scenarios in mTEDx. Additionally, we find that SegAugment is also applicable to purely sentence-level data, as in CoVoST, and that it enables Speech Translation models to completely close the gap between the gold and automatic segmentation at inference time.

字幕（替代）的语音翻译是通过将符合特定显示指南的字幕划分插入字幕中断，将语音数据自动转化为良好的字幕。与语音翻译（ST）类似，模型训练需要并行数据，其中包括音频输入与其文本翻译配对。然而，在替代方面，还必须用字幕断裂来注释文本。到目前为止，这一要求代表了系统开发的瓶颈，如公开可用的替代公司所证实。为了填补这一空白，我们提出了一种在不干预的情况下将现有的ST Corpora转换为替代资源的方法。我们构建了一个分段模型，该模型通过以多模式的方式利用音频和文本来自动将文本片段分为适当的字幕，从而在零拍摄条件下实现了高分子的质量。对手动和自动分割培训的替代系统的比较实验导致相似的性能，显示了我们方法的有效性。

We present SpeechMatrix, a large-scale multilingual corpus of speech-to-speech translations mined from real speech of European Parliament recordings. It contains speech alignments in 136 language pairs with a total of 418 thousand hours of speech. To evaluate the quality of this parallel speech, we train bilingual speech-to-speech translation models on mined data only and establish extensive baseline results on EuroParl-ST, VoxPopuli and FLEURS test sets. Enabled by the multilinguality of SpeechMatrix, we also explore multilingual speech-to-speech translation, a topic which was addressed by few other works. We also demonstrate that model pre-training and sparse scaling using Mixture-of-Experts bring large gains to translation performance. The mined data and models are freely available.

语音细分将长言语分为短段，对于语音翻译（ST）至关重要。像WebRTC VAD这样的流行VAD工具通常依赖于基于暂停的细分。不幸的是，语音中的暂停不一定与句子边界匹配，句子可以通过很短的停顿连接，而VAD很难检测到。在这项研究中，我们建议使用使用分割的双语语音语料库训练的二元分类模型进行语音分割方法。我们还提出了一种结合VAD和上述语音分割方法的混合方法。实验结果表明，所提出的方法比常规分割方法更适合级联和端到端ST系统。混合方法进一步改善了翻译性能。

End-to-end Speech Translation (E2E ST) aims to translate source speech into target translation without generating the intermediate transcript. However, existing approaches for E2E ST degrade considerably when only limited ST data are available. We observe that an ST model's performance strongly correlates with its embedding similarity from speech and transcript. In this paper, we propose Word-Aligned COntrastive learning (WACO), a novel method for few-shot speech-to-text translation. Our key idea is bridging word-level representations for both modalities via contrastive learning. We evaluate WACO and other methods on the MuST-C dataset, a widely used ST benchmark. Our experiments demonstrate that WACO outperforms the best baseline methods by 0.7-8.5 BLEU points with only 1-hour parallel data. Code is available at https://anonymous.4open.science/r/WACO .

我们介绍了一种无线文字语音转换（S2ST）系统，可以将来自一种语言的语音转换为另一种语言，并且可以在不需要任何文本数据的情况下构建。与文献中的现有工作不同，我们解决了模拟多扬声器目标语音的挑战，并用现实世界的S2ST数据训练系统。我们方法的关键是一种自我监督的单位语音标准化技术，该标准化技术将预先训练的语音编码器具有来自多个扬声器的配对声音，以及单个参考扬声器，以减少由于复印件引起的变化，同时保留词汇内容。只有10分钟的语音标准化的配对数据，我们在培训\ vp〜s2st数据集上的S2ST模型时获得平均3.2 BLEU增益，而不是在未标准化的语音目标上培训的基线。我们还将自动开采的S2ST数据纳入并显示额外的2.0 BLEU增益。据我们所知，我们是第一个建立无线的S2ST技术，可以用真实世界的数据培训，并为多种语言配对工作。

Direct speech-to-speech translation (S2ST), in which all components can be optimized jointly, is advantageous over cascaded approaches to achieve fast inference with a simplified pipeline. We present a novel two-pass direct S2ST architecture, {\textit UnitY}, which first generates textual representations and predicts discrete acoustic units subsequently. We enhance the model performance by subword prediction in the first-pass decoder, advanced two-pass decoder architecture design and search strategy, and better training regularization. To leverage large amounts of unlabeled text data, we pre-train the first-pass text decoder based on the self-supervised denoising auto-encoding task. Experimental evaluations on benchmark datasets at various data scales demonstrate that UnitY outperforms a single-pass speech-to-unit translation model by 2.5-4.2 ASR-BLEU with 2.83x decoding speed-up. We show that the proposed methods boost the performance even when predicting spectrogram in the second pass. However, predicting discrete units achieves 2.51x decoding speed-up compared to that case.

In this paper, we introduce our work of building a Streaming Multilingual Speech Model (SM2), which can transcribe or translate multiple spoken languages into texts of the target language. The backbone of SM2 is Transformer Transducer, which has high streaming capability. Instead of human labeled speech translation (ST) data, SM2 models are trained using weakly supervised data generated by converting the transcriptions in speech recognition corpora with a machine translation service. With 351 thousand hours of anonymized speech training data from 25 languages, SM2 models achieve comparable or even better ST quality than some recent popular large-scale non-streaming speech models. More importantly, we show that SM2 has the truly zero-shot capability when expanding to new target languages, yielding high quality ST results for {source-speech, target-text} pairs that are not seen during training.

自动副标题是将视听产品的语音自动转化为短文本的任务，换句话说，字幕及其相应的时间戳。生成的字幕需要符合多个空间和时间要求（长度，阅读速度），同时与语音同步并以促进理解的方式进行分割。鉴于其相当大的复杂性，迄今为止，通过分别处理转录，翻译，分割为字幕并预测时间戳的元素来解决自动字幕。在本文中，我们提出了第一个直接自动字幕模型，该模型在单个解决方案中从源语音中生成目标语言字幕及其时间戳。与经过内外数据和外域数据训练的最先进的级联模型的比较表明，我们的系统提供了高质量的字幕，同时在整合性方面也具有竞争力，并具有维护单个模型的所有优势。

直接语音到语音翻译（S2ST）模型与传统级联系统可用的数据量相比，几乎没有平行的S2ST数据遇到数据稀缺问题，该数据包括自动语音识别（ASR），机器翻译（MT）和文本到语音（TTS）合成。在这项工作中，我们使用未标记的语音数据和数据扩展来探索自我监督的预训练，以解决此问题。我们利用了最近提出的语音到单位翻译（S2UT）框架，该框架将目标语音编码为离散表示形式，并转移前训练前和有效的部分填充技术，可很好地适用于语音到文本翻译（S2T）通过研究语音编码器和离散单位解码器预训练，S2UT域。我们在西班牙语 - 英语翻译上进行的实验表明，与多任务学习相比，自我监督的预训练始终如一地提高模型性能，平均为6.6-12.1 BLEU增长，并且可以与数据增强技术相结合，以应用MT来创建弱监督监督的培训数据。音频样本可在以下网址获得：https：//facebookresearch.github.io/speech_translation/enhanced_direct_s2st_units/index.html。

How to solve the data scarcity problem for end-to-end speech-to-text translation (ST)? It's well known that data augmentation is an efficient method to improve performance for many tasks by enlarging the dataset. In this paper, we propose Mix at three levels for Speech Translation (M^3ST) method to increase the diversity of the augmented training corpus. Specifically, we conduct two phases of fine-tuning based on a pre-trained model using external machine translation (MT) data. In the first stage of fine-tuning, we mix the training corpus at three levels, including word level, sentence level and frame level, and fine-tune the entire model with mixed data. At the second stage of fine-tuning, we take both original speech sequences and original text sequences in parallel into the model to fine-tune the network, and use Jensen-Shannon divergence to regularize their outputs. Experiments on MuST-C speech translation benchmark and analysis show that M^3ST outperforms current strong baselines and achieves state-of-the-art results on eight directions with an average BLEU of 29.9.

端到端（E2E）语音到文本翻译（ST）通常取决于通过语音识别或文本翻译任务使用源成绩单预处理其编码器和/或解码器，否则翻译性能会大大下降。但是，笔录并不总是可用的，在文献中很少研究这种预处理的E2E ST。在本文中，我们重新审视了这个问题，并探讨了仅在语音翻译对培训的E2E ST质量的程度。我们重新审查了几种证明对ST的有益的技术，并提供了一系列最佳实践，这些实践使基于变压器的E2E ST系统偏向于从头开始训练。此外，我们提出了参数化的距离惩罚，以促进语音自我注意模型中的位置建模。在涵盖23种语言的四个基准测试中，我们的实验表明，在不使用任何成绩单或预处理的情况下，提议的系统达到甚至优于先前采用预处理的研究，尽管差距仍然存在（极为）低资源的设置。最后，我们讨论了神经声学特征建模，其中神经模型旨在直接从原始语音信号中提取声学特征，以简化电感偏见并为模型描述语音增添自由度。我们第一次证明了它的可行性，并在ST任务上表现出令人鼓舞的结果。

本文介绍了流媒体和非流定向晶体翻译的统一端到端帧工作。虽然非流媒体语音翻译的培训配方已经成熟，但尚未建立流媒体传播的食谱。在这项工作中，WEFOCUS在开发一个统一的模型（UNIST），它从基本组成部分的角度支持流媒体和非流媒体ST，包括培训目标，注意机制和解码政策。对最流行的语音到文本翻译基准数据集，MERE-C的实验表明，与媒体ST的BLEU评分和延迟度量有更好的折衷和液化标准端到端基线和级联模型。我们将公开提供我们的代码和评估工具。

End-to-End speech-to-speech translation (S2ST) is generally evaluated with text-based metrics. This means that generated speech has to be automatically transcribed, making the evaluation dependent on the availability and quality of automatic speech recognition (ASR) systems. In this paper, we propose a text-free evaluation metric for end-to-end S2ST, named BLASER, to avoid the dependency on ASR systems. BLASER leverages a multilingual multimodal encoder to directly encode the speech segments for source input, translation output and reference into a shared embedding space and computes a score of the translation quality that can be used as a proxy to human evaluation. To evaluate our approach, we construct training and evaluation sets from more than 40k human annotations covering seven language directions. The best results of BLASER are achieved by training with supervision from human rating scores. We show that when evaluated at the sentence level, BLASER correlates significantly better with human judgment compared to ASR-dependent metrics including ASR-SENTBLEU in all translation directions and ASR-COMET in five of them. Our analysis shows combining speech and text as inputs to BLASER does not increase the correlation with human scores, but best correlations are achieved when using speech, which motivates the goal of our research. Moreover, we show that using ASR for references is detrimental for text-based metrics.

本文介绍了我们针对IWSLT 2022离线任务的端到端Yitrans语音翻译系统的提交，该任务从英语音频转换为德语，中文和日语。 Yitrans系统建立在大规模训练的编码器模型上。更具体地说，我们首先设计了多阶段的预训练策略，以建立具有大量标记和未标记数据的多模式模型。然后，我们为下游语音翻译任务微调模型的相应组件。此外，我们做出了各种努力，以提高性能，例如数据过滤，数据增强，语音细分，模型集合等。实验结果表明，我们的Yitrans系统比在三个翻译方向上的强基线取得了显着改进，并且比去年在TST2021英语 - 德国人中的最佳端到端系统方面的改进+5.2 BLEU改进。根据自动评估指标，我们的最终意见在英语 - 德国和英语端到端系统上排名第一。我们使代码和模型公开可用。

Speech translation (ST) is the task of directly translating acoustic speech signals in a source language into text in a foreign language. ST task has been addressed, for a long time, using a pipeline approach with two modules : first an Automatic Speech Recognition (ASR) in the source language followed by a text-to-text Machine translation (MT). In the past few years, we have seen a paradigm shift towards the end-to-end approaches using sequence-to-sequence deep neural network models. This paper presents our efforts towards the development of the first Broadcast News end-to-end Arabic to English speech translation system. Starting from independent ASR and MT LDC releases, we were able to identify about 92 hours of Arabic audio recordings for which the manual transcription was also translated into English at the segment level. These data was used to train and compare pipeline and end-to-end speech translation systems under multiple scenarios including transfer learning and data augmentation techniques.

本文介绍了基于Wav2VEC 2.0的跨语言语音表示学习的大规模模型。我们在128种语言中培训最多2B个公共讲话音频的近半小时的型号的模型，比公共数据的数量级比最大的已知事先工作。我们的评估涵盖了广泛的任务，域，数据制度和语言，都是高低资源。在Covost-2语音翻译基准测试中，我们将先前的最先进的状态平均为7.4 BLEU超过21个翻译方向进入英语。对于语音识别，XLS-R在Babel，MLS，CommonVoice以及Voxpopuli上的最佳已知工作中提高，降低了相对的误差率14-34％。 XLS-R还在Voxlingua107语言识别上设置了新的技术状态。此外，我们表明，具有足够的模型规模，交叉思维预先预测可以在将英语演讲翻译成其他语言时才能优于英语撇印，这是一个有利于单晶的预借预制的设置。我们希望XLS-R可以帮助改善世界上更多语言的语音处理任务。

To alleviate the data scarcity problem in End-to-end speech translation (ST), pre-training on data for speech recognition and machine translation is considered as an important technique. However, the modality gap between speech and text prevents the ST model from efficiently inheriting knowledge from the pre-trained models. In this work, we propose AdaTranS for end-to-end ST. It adapts the speech features with a new shrinking mechanism to mitigate the length mismatch between speech and text features by predicting word boundaries. Experiments on the MUST-C dataset demonstrate that AdaTranS achieves better performance than the other shrinking-based methods, with higher inference speed and lower memory usage. Further experiments also show that AdaTranS can be equipped with additional alignment losses to further improve performance.

在本文中，我们介绍了一个高质量的大规模基准数据集，用于英语 - 越南语音翻译，其中有508音频小时，由331k的三胞胎组成（句子长度的音频，英语源笔录句，越南人目标subtitle句子）。我们还使用强基础进行了经验实验，发现传统的“级联”方法仍然优于现代“端到端”方法。据我们所知，这是第一个大规模的英语 - 越南语音翻译研究。我们希望我们的公开数据集和研究都可以作为未来研究和英语语音翻译应用的起点。我们的数据集可从https://github.com/vinairesearch/phost获得