Video dubbing aims to translate the original speech in a film or television program into the speech in a target language, which can be achieved with a cascaded system consisting of speech recognition, machine translation and speech synthesis. To ensure the translated speech to be well aligned with the corresponding video, the length/duration of the translated speech should be as close as possible to that of the original speech, which requires strict length control. Previous works usually control the number of words or characters generated by the machine translation model to be similar to the source sentence, without considering the isochronicity of speech as the speech duration of words/characters in different languages varies. In this paper, we propose a machine translation system tailored for the task of video dubbing, which directly considers the speech duration of each token in translation, to match the length of source and target speech. Specifically, we control the speech length of generated sentence by guiding the prediction of each word with the duration information, including the speech duration of itself as well as how much duration is left for the remaining words. We design experiments on four language directions (German -> English, Spanish -> English, Chinese <-> English), and the results show that the proposed method achieves better length control ability on the generated speech than baseline methods. To make up the lack of real-world datasets, we also construct a real-world test set collected from films to provide comprehensive evaluations on the video dubbing task.
translated by 谷歌翻译
我们介绍了Prosody-Aware Machine翻译的任务,旨在产生适合配音的翻译。配音是口语句要求将内容传输以及源的韵律结构转移到目标语言中以保留时序信息。实际上,这意味着从源暂停到目标并确保目标语音段具有大致相同的源片段的暂停。在这项工作中,我们提出了一种隐含和明确的建模方法,将韵律信息整合到神经机翻译中。英语 - 德语/法语与自动指标的实验表明,最简单的考虑方法最佳。结果是通过人类评估的翻译和配音视频确认。
translated by 谷歌翻译
在本文中,我们提出了一个神经端到端系统,用于保存视频的语音,唇部同步翻译。该系统旨在将多个组件模型结合在一起,并以目标语言的目标语言与目标语言的原始扬声器演讲的视频与目标语音相结合,但在语音,语音特征,面对原始扬声器的视频中保持着重点。管道从自动语音识别开始,包括重点检测,然后是翻译模型。然后,翻译后的文本由文本到语音模型合成,该模型重新创建了原始句子映射的原始重点。然后,使用语音转换模型将结果的合成语音映射到原始扬声器的声音。最后,为了将扬声器的嘴唇与翻译的音频同步,有条件的基于对抗网络的模型生成了相对于输入面图像以及语音转换模型的输出的适应性唇部运动的帧。最后,系统将生成的视频与转换后的音频结合在一起,以产生最终输出。结果是一个扬声器用另一种语言说话的视频而不真正知道。为了评估我们的设计,我们介绍了完整系统的用户研究以及对单个组件的单独评估。由于没有可用的数据集来评估我们的整个系统,因此我们收集了一个测试集并在此测试集上评估我们的系统。结果表明,我们的系统能够生成令人信服的原始演讲者的视频,同时保留原始说话者的特征。收集的数据集将共享。
translated by 谷歌翻译
We present SpeechMatrix, a large-scale multilingual corpus of speech-to-speech translations mined from real speech of European Parliament recordings. It contains speech alignments in 136 language pairs with a total of 418 thousand hours of speech. To evaluate the quality of this parallel speech, we train bilingual speech-to-speech translation models on mined data only and establish extensive baseline results on EuroParl-ST, VoxPopuli and FLEURS test sets. Enabled by the multilinguality of SpeechMatrix, we also explore multilingual speech-to-speech translation, a topic which was addressed by few other works. We also demonstrate that model pre-training and sparse scaling using Mixture-of-Experts bring large gains to translation performance. The mined data and models are freely available.
translated by 谷歌翻译
中文方言文本到语音(TTS)系统通常只能由本地语言学家使用,因为中文方言的书面形式具有不同的字符,成语,语法和使用普通话,甚至本地扬声器也无法输入正确的句子。对于普通话的文本输入,中国方言TT只能产生部分挑剔的语音,而韵律和自然性相对较差。为了降低使用栏并使其在商业广告中更实用,我们提出了一种新型的中国方言TTS前端,并带有翻译模块。它有助于使用正确的拼字法和语法将普通话文本转换为惯用表达式,以便可以改善合成语音的清晰度和自然性。为翻译任务提出了一种具有浏览抽样策略的非自动入围神经机器翻译模型。这是将翻译与TTS Frontend合并的第一项已知作品。我们对广东话的实验批准,拟议的前端可以帮助广东TTS系统通过普通话输入来提高0.27的MOS。
translated by 谷歌翻译
语音到语音翻译(S2ST)将输入语音转换为另一种语言。实时交付S2ST的挑战是翻译和语音合成模块之间的累积延迟。尽管最近增量的文本到语音(ITTS)模型已显示出巨大的质量改进,但它们通常需要其他未来的文本输入才能达到最佳性能。在这项工作中,我们通过调整上游语音翻译器来为语音合成器生成高质量的伪lookahead来最大程度地减少ITT的最初等待时间。缓解初始延迟后,我们证明了合成语音的持续时间在延迟中也起着至关重要的作用。我们将其形式化为延迟度量,然后提出一种简单而有效的持续时间缩放方法,以减少延迟。我们的方法始终将延迟减少0.2-0.5秒,而无需牺牲语音翻译质量。
translated by 谷歌翻译
由于其有条件的独立性假设,非自动回忆翻译(NAT)模型很难捕获目标翻译的多模式分布,这被称为“多模式性问题”,包括词汇多模式和句法。多模式。虽然对第一个进行了充分的研究,但句法多模式性为NAT的标准横熵(XE)损失带来了严重的挑战,并且正在研究中。在本文中,我们对句法多模式问题进行了系统研究。具体而言,我们将其分解为短期和远程句法多模式,并在精心设计的合成数据集和真实数据集上评估了具有高级损耗函数的几种NAT算法。我们发现,连接派时间分类(CTC)损失和订单不合时宜的熵(OAXE)损失可以更好地处理短期和远程语法多模式。此外,我们将同时掌握并设计新的损失功能,以更好地处理现实世界数据集中复杂的句法多模式。为了促进实际用法,我们提供了一个指南,以使用不同种类的句法多模式的不同损失功能。
translated by 谷歌翻译
我们呈现TranslatOrron 2,一个神经直接语音转换转换模型,可以训练结束到底。 TranslatOrron 2由语音编码器,音素解码器,MEL谱图合成器和连接所有前三个组件的注意模块组成。实验结果表明,翻译ron 2在翻译质量和预测的语音自然方面,通过大幅度优于原始翻译,并且通过减轻超越,例如唠叨或长暂停来大幅提高预测演讲的鲁棒性。我们还提出了一种在翻译语音中保留源代言人声音的新方法。训练有素的模型被限制为保留源扬声器的声音,但与原始翻译ron不同,它无法以不同的扬声器的语音产生语音,使模型对生产部署更加强大,通过减轻潜在的滥用来创建欺骗音频伪影。当新方法与基于简单的替代的数据增强一起使用时,训练的翻译器2模型能够保留每个扬声器的声音,以便用扬声器转动输入输入。
translated by 谷歌翻译
自动副标题是将视听产品的语音自动转化为短文本的任务,换句话说,字幕及其相应的时间戳。生成的字幕需要符合多个空间和时间要求(长度,阅读速度),同时与语音同步并以促进理解的方式进行分割。鉴于其相当大的复杂性,迄今为止,通过分别处理转录,翻译,分割为字幕并预测时间戳的元素来解决自动字幕。在本文中,我们提出了第一个直接自动字幕模型,该模型在单个解决方案中从源语音中生成目标语言字幕及其时间戳。与经过内外数据和外域数据训练的最先进的级联模型的比较表明,我们的系统提供了高质量的字幕,同时在整合性方面也具有竞争力,并具有维护单个模型的所有优势。
translated by 谷歌翻译
我们提出了直接同时的语音转换(SIMUL-S2ST)模型,此外,翻译的产生与中间文本表示无关。我们的方法利用了最近与离散单位直接语音转换的最新进展,其中从模型中预测了一系列离散表示,而不是连续频谱图特征,而不是以无监督的方式学习,并直接传递给语音的声码器综合在一起。我们还介绍了变分单调的多口语注意力(V-MMA),以处理语音同声翻译中效率低效的政策学习的挑战。然后,同时策略在源语音特征和目标离散单元上运行。我们开展实证研究,比较级联和直接方法对Fisher西班牙语 - 英语和必需的英语西班牙语数据集。直接同步模型显示通过在翻译质量和延迟之间实现更好的权衡来优于级联模型。
translated by 谷歌翻译
本文介绍了一种新的数据增强方法,用于神经机器翻译,该方法可以在语言内部和跨语言内部实施更强的语义一致性。我们的方法基于条件掩盖语言模型(CMLM),该模型是双向的,可以在左右上下文以及标签上有条件。我们证明CMLM是生成上下文依赖性单词分布的好技术。特别是,我们表明CMLM能够通过在替换过程中对源和目标进行调节来实现语义一致性。此外,为了增强多样性,我们将软词替换的想法纳入了数据增强,该概念用词汇上的概率分布代替了一个单词。在不同量表的四个翻译数据集上进行的实验表明,总体解决方案会导致更现实的数据增强和更好的翻译质量。与最新作品相比,我们的方法始终取得了最佳性能,并且在基线上的提高了1.90个BLEU点。
translated by 谷歌翻译
Data scarcity is one of the main issues with the end-to-end approach for Speech Translation, as compared to the cascaded one. Although most data resources for Speech Translation are originally document-level, they offer a sentence-level view, which can be directly used during training. But this sentence-level view is single and static, potentially limiting the utility of the data. Our proposed data augmentation method SegAugment challenges this idea and aims to increase data availability by providing multiple alternative sentence-level views of a dataset. Our method heavily relies on an Audio Segmentation system to re-segment the speech of each document, after which we obtain the target text with alignment methods. The Audio Segmentation system can be parameterized with different length constraints, thus giving us access to multiple and diverse sentence-level views for each document. Experiments in MuST-C show consistent gains across 8 language pairs, with an average increase of 2.2 BLEU points, and up to 4.7 BLEU for lower-resource scenarios in mTEDx. Additionally, we find that SegAugment is also applicable to purely sentence-level data, as in CoVoST, and that it enables Speech Translation models to completely close the gap between the gold and automatic segmentation at inference time.
translated by 谷歌翻译
在本文中,我们介绍了一个高质量的大规模基准数据集,用于英语 - 越南语音翻译,其中有508音频小时,由331k的三胞胎组成(句子长度的音频,英语源笔录句,越南人目标subtitle句子)。我们还使用强基础进行了经验实验,发现传统的“级联”方法仍然优于现代“端到端”方法。据我们所知,这是第一个大规模的英语 - 越南语音翻译研究。我们希望我们的公开数据集和研究都可以作为未来研究和英语语音翻译应用的起点。我们的数据集可从https://github.com/vinairesearch/phost获得
translated by 谷歌翻译
Directly training a document-to-document (Doc2Doc) neural machine translation (NMT) via Transformer from scratch, especially on small datasets usually fails to converge. Our dedicated probing tasks show that 1) both the absolute position and relative position information gets gradually weakened or even vanished once it reaches the upper encoder layers, and 2) the vanishing of absolute position information in encoder output causes the training failure of Doc2Doc NMT. To alleviate this problem, we propose a position-aware Transformer (P-Transformer) to enhance both the absolute and relative position information in both self-attention and cross-attention. Specifically, we integrate absolute positional information, i.e., position embeddings, into the query-key pairs both in self-attention and cross-attention through a simple yet effective addition operation. Moreover, we also integrate relative position encoding in self-attention. The proposed P-Transformer utilizes sinusoidal position encoding and does not require any task-specified position embedding, segment embedding, or attention mechanism. Through the above methods, we build a Doc2Doc NMT model with P-Transformer, which ingests the source document and completely generates the target document in a sequence-to-sequence (seq2seq) way. In addition, P-Transformer can be applied to seq2seq-based document-to-sentence (Doc2Sent) and sentence-to-sentence (Sent2Sent) translation. Extensive experimental results of Doc2Doc NMT show that P-Transformer significantly outperforms strong baselines on widely-used 9 document-level datasets in 7 language pairs, covering small-, middle-, and large-scales, and achieves a new state-of-the-art. Experimentation on discourse phenomena shows that our Doc2Doc NMT models improve the translation quality in both BLEU and discourse coherence. We make our code available on Github.
translated by 谷歌翻译
End-to-End speech-to-speech translation (S2ST) is generally evaluated with text-based metrics. This means that generated speech has to be automatically transcribed, making the evaluation dependent on the availability and quality of automatic speech recognition (ASR) systems. In this paper, we propose a text-free evaluation metric for end-to-end S2ST, named BLASER, to avoid the dependency on ASR systems. BLASER leverages a multilingual multimodal encoder to directly encode the speech segments for source input, translation output and reference into a shared embedding space and computes a score of the translation quality that can be used as a proxy to human evaluation. To evaluate our approach, we construct training and evaluation sets from more than 40k human annotations covering seven language directions. The best results of BLASER are achieved by training with supervision from human rating scores. We show that when evaluated at the sentence level, BLASER correlates significantly better with human judgment compared to ASR-dependent metrics including ASR-SENTBLEU in all translation directions and ASR-COMET in five of them. Our analysis shows combining speech and text as inputs to BLASER does not increase the correlation with human scores, but best correlations are achieved when using speech, which motivates the goal of our research. Moreover, we show that using ASR for references is detrimental for text-based metrics.
translated by 谷歌翻译
The word alignment task, despite its prominence in the era of statistical machine translation (SMT), is niche and under-explored today. In this two-part tutorial, we argue for the continued relevance for word alignment. The first part provides a historical background to word alignment as a core component of the traditional SMT pipeline. We zero-in on GIZA++, an unsupervised, statistical word aligner with surprising longevity. Jumping forward to the era of neural machine translation (NMT), we show how insights from word alignment inspired the attention mechanism fundamental to present-day NMT. The second part shifts to a survey approach. We cover neural word aligners, showing the slow but steady progress towards surpassing GIZA++ performance. Finally, we cover the present-day applications of word alignment, from cross-lingual annotation projection, to improving translation.
translated by 谷歌翻译
我们介绍了一种无线文字语音转换(S2ST)系统,可以将来自一种语言的语音转换为另一种语言,并且可以在不需要任何文本数据的情况下构建。与文献中的现有工作不同,我们解决了模拟多扬声器目标语音的挑战,并用现实世界的S2ST数据训练系统。我们方法的关键是一种自我监督的单位语音标准化技术,该标准化技术将预先训练的语音编码器具有来自多个扬声器的配对声音,以及单个参考扬声器,以减少由于复印件引起的变化,同时保留词汇内容。只有10分钟的语音标准化的配对数据,我们在培训\ vp〜s2st数据集上的S2ST模型时获得平均3.2 BLEU增益,而不是在未标准化的语音目标上培训的基线。我们还将自动开采的S2ST数据纳入并显示额外的2.0 BLEU增益。据我们所知,我们是第一个建立无线的S2ST技术,可以用真实世界的数据培训,并为多种语言配对工作。
translated by 谷歌翻译
我们介绍了CVSS,这是一种大规模的多语言对语音转换(S2ST)语料库,从21种语言覆盖了21种语言的句子级并行S2ST对。通过将Covost 2从Covost 2的翻译文本综合将翻译文本与最先进的TTS系统合成语音,源自公共语音语音语料库和COVOST 2语音到文本转换(ST)语料库。提供了两个版本的翻译演讲:1)CVSS-C:所有翻译演讲都是一种高质量的规范声音; 2)CVSS-T:翻译语音从相应的源语音传输。此外,CVSS提供标准化的翻译文本,它与翻译语音中的发音匹配。在每个版本的CVSS上,我们建立了基线多语言直接S2ST模型和Cascade S2ST模型,验证了语料库的有效性。为了构建强大的Cascade S2ST基准,我们在Covost 2上培训了St模型,这优于前一种最先进的培训,而无需额外的数据。尽管如此,直接S2ST模型的性能在从头开始训练时接近强级联基线,并且在匹配ST模型中初始化时,仅在ASR转换转换时的0.1或0.7bleu差异。
translated by 谷歌翻译
本文介绍了流媒体和非流定向晶体翻译的统一端到端帧工作。虽然非流媒体语音翻译的培训配方已经成熟,但尚未建立流媒体传播的食谱。在这项工作中,WEFOCUS在开发一个统一的模型(UNIST),它从基本组成部分的角度支持流媒体和非流媒体ST,包括培训目标,注意机制和解码政策。对最流行的语音到文本翻译基准数据集,MERE-C的实验表明,与媒体ST的BLEU评分和延迟度量有更好的折衷和液化标准端到端基线和级联模型。我们将公开提供我们的代码和评估工具。
translated by 谷歌翻译
Error correction techniques have been used to refine the output sentences from automatic speech recognition (ASR) models and achieve a lower word error rate (WER) than original ASR outputs. Previous works usually use a sequence-to-sequence model to correct an ASR output sentence autoregressively, which causes large latency and cannot be deployed in online ASR services. A straightforward solution to reduce latency, inspired by non-autoregressive (NAR) neural machine translation, is to use an NAR sequence generation model for ASR error correction, which, however, comes at the cost of significantly increased ASR error rate. In this paper, observing distinctive error patterns and correction operations (i.e., insertion, deletion, and substitution) in ASR, we propose FastCorrect, a novel NAR error correction model based on edit alignment. In training, FastCorrect aligns each source token from an ASR output sentence to the target tokens from the corresponding ground-truth sentence based on the edit distance between the source and target sentences, and extracts the number of target tokens corresponding to each source token during edition/correction, which is then used to train a length predictor and to adjust the source tokens to match the length of the target sentence for parallel generation. In inference, the token number predicted by the length predictor is used to adjust the source tokens for target sequence generation. Experiments on the public AISHELL-1 dataset and an internal industrial-scale ASR dataset show the effectiveness of FastCorrect for ASR error correction: 1) it speeds up the inference by 6-9 times and maintains the accuracy (8-14% WER reduction) compared with the autoregressive correction model; and 2) it outperforms the popular NAR models adopted in neural machine translation and text edition by a large margin.
translated by 谷歌翻译