The black-box nature of end-to-end speech translation (E2E ST) systems makes it difficult to understand how source language inputs are being mapped to the target language. To solve this problem, we would like to simultaneously generate automatic speech recognition (ASR) and ST predictions such that each source language word is explicitly mapped to a target language word. A major challenge arises from the fact that translation is a non-monotonic sequence transduction task due to word ordering differences between languages -- this clashes with the monotonic nature of ASR. Therefore, we propose to generate ST tokens out-of-order while remembering how to re-order them later. We achieve this by predicting a sequence of tuples consisting of a source word, the corresponding target words, and post-editing operations dictating the correct insertion points for the target word. We examine two variants of such operation sequences which enable generation of monotonic transcriptions and non-monotonic translations from the same speech input simultaneously. We apply our approach to offline and real-time streaming models, demonstrating that we can provide explainable translations without sacrificing quality or latency. In fact, the delayed re-ordering ability of our approach improves performance during streaming. As an added benefit, our method performs ASR and ST simultaneously, making it faster than using two separate systems to perform these tasks.

本文介绍了流媒体和非流定向晶体翻译的统一端到端帧工作。虽然非流媒体语音翻译的培训配方已经成熟，但尚未建立流媒体传播的食谱。在这项工作中，WEFOCUS在开发一个统一的模型（UNIST），它从基本组成部分的角度支持流媒体和非流媒体ST，包括培训目标，注意机制和解码政策。对最流行的语音到文本翻译基准数据集，MERE-C的实验表明，与媒体ST的BLEU评分和延迟度量有更好的折衷和液化标准端到端基线和级联模型。我们将公开提供我们的代码和评估工具。

我们提出了直接同时的语音转换（SIMUL-S2ST）模型，此外，翻译的产生与中间文本表示无关。我们的方法利用了最近与离散单位直接语音转换的最新进展，其中从模型中预测了一系列离散表示，而不是连续频谱图特征，而不是以无监督的方式学习，并直接传递给语音的声码器综合在一起。我们还介绍了变分单调的多口语注意力（V-MMA），以处理语音同声翻译中效率低效的政策学习的挑战。然后，同时策略在源语音特征和目标离散单元上运行。我们开展实证研究，比较级联和直接方法对Fisher西班牙语 - 英语和必需的英语西班牙语数据集。直接同步模型显示通过在翻译质量和延迟之间实现更好的权衡来优于级联模型。

Data scarcity is one of the main issues with the end-to-end approach for Speech Translation, as compared to the cascaded one. Although most data resources for Speech Translation are originally document-level, they offer a sentence-level view, which can be directly used during training. But this sentence-level view is single and static, potentially limiting the utility of the data. Our proposed data augmentation method SegAugment challenges this idea and aims to increase data availability by providing multiple alternative sentence-level views of a dataset. Our method heavily relies on an Audio Segmentation system to re-segment the speech of each document, after which we obtain the target text with alignment methods. The Audio Segmentation system can be parameterized with different length constraints, thus giving us access to multiple and diverse sentence-level views for each document. Experiments in MuST-C show consistent gains across 8 language pairs, with an average increase of 2.2 BLEU points, and up to 4.7 BLEU for lower-resource scenarios in mTEDx. Additionally, we find that SegAugment is also applicable to purely sentence-level data, as in CoVoST, and that it enables Speech Translation models to completely close the gap between the gold and automatic segmentation at inference time.

The study of the attention mechanism has sparked interest in many fields, such as language modeling and machine translation. Although its patterns have been exploited to perform different tasks, from neural network understanding to textual alignment, no previous work has analysed the encoder-decoder attention behavior in speech translation (ST) nor used it to improve ST on a specific task. In this paper, we fill this gap by proposing an attention-based policy (EDAtt) for simultaneous ST (SimulST) that is motivated by an analysis of the existing attention relations between audio input and textual output. Its goal is to leverage the encoder-decoder attention scores to guide inference in real time. Results on en->{de, es} show that the EDAtt policy achieves overall better results compared to the SimulST state of the art, especially in terms of computational-aware latency.

Direct speech-to-speech translation (S2ST), in which all components can be optimized jointly, is advantageous over cascaded approaches to achieve fast inference with a simplified pipeline. We present a novel two-pass direct S2ST architecture, {\textit UnitY}, which first generates textual representations and predicts discrete acoustic units subsequently. We enhance the model performance by subword prediction in the first-pass decoder, advanced two-pass decoder architecture design and search strategy, and better training regularization. To leverage large amounts of unlabeled text data, we pre-train the first-pass text decoder based on the self-supervised denoising auto-encoding task. Experimental evaluations on benchmark datasets at various data scales demonstrate that UnitY outperforms a single-pass speech-to-unit translation model by 2.5-4.2 ASR-BLEU with 2.83x decoding speed-up. We show that the proposed methods boost the performance even when predicting spectrogram in the second pass. However, predicting discrete units achieves 2.51x decoding speed-up compared to that case.

最先进的编码器模型（例如，用于机器翻译（MT）或语音识别（ASR））作为原子单元构造并端到端训练。没有其他模型的任何组件都无法（重新）使用。我们描述了Legonn，这是一种使用解码器模块构建编码器架构的过程，可以在各种MT和ASR任务中重复使用，而无需进行任何微调。为了实现可重复性，每个编码器和解码器模块之间的界面都基于模型设计器预先定义的离散词汇，将其接地到边缘分布序列。我们提出了两种摄入这些边缘的方法。一个是可区分的，可以使整个网络的梯度流动，另一个是梯度分离的。为了使MT任务之间的解码器模块的可移植性用于不同的源语言和其他任务（例如ASR），我们引入了一种模态不可思议的编码器，该模态编码器由长度控制机制组成，以动态调整编码器的输出长度，以匹配预期的输入长度范围的范围预训练的解码器。我们提出了几项实验来证明Legonn模型的有效性：可以重复使用德国英语（DE-EN）MT任务的训练有素的语言解码器模块，而没有对Europarl English ASR和ROMANIAN-ENGLISH进行微调（RO）（RO）（RO）（RO） -en）MT任务以匹配或击败相应的基线模型。当针对数千个更新的目标任务进行微调时，我们的Legonn模型将RO-EN MT任务提高了1.5个BLEU点，并为Europarl ASR任务降低了12.5％的相对减少。此外，为了显示其可扩展性，我们从三个模块中构成了一个legonn ASR模型 - 每个模块都在三个不同数据集的不同端到端训练的模型中学习 - 将降低的减少降低到19.5％。

In this paper, we introduce our work of building a Streaming Multilingual Speech Model (SM2), which can transcribe or translate multiple spoken languages into texts of the target language. The backbone of SM2 is Transformer Transducer, which has high streaming capability. Instead of human labeled speech translation (ST) data, SM2 models are trained using weakly supervised data generated by converting the transcriptions in speech recognition corpora with a machine translation service. With 351 thousand hours of anonymized speech training data from 25 languages, SM2 models achieve comparable or even better ST quality than some recent popular large-scale non-streaming speech models. More importantly, we show that SM2 has the truly zero-shot capability when expanding to new target languages, yielding high quality ST results for {source-speech, target-text} pairs that are not seen during training.

神经传感器已被广泛用于自动语音识别（ASR）。在本文中，我们将其介绍给流端到端语音翻译（ST），该语音旨在将音频信号直接转换为其他语言的文本。与执行ASR之后的级联ST相比，基于文本的机器翻译（MT），拟议的变压器传感器（TT）基于ST模型大大降低了推理潜伏期，利用语音信息并避免了从ASR到MT的错误传播。为了提高建模能力，我们提出了TT中联合网络的注意集合。此外，我们将基于TT的ST扩展到多语言ST，该ST同时生成多种语言的文本。大规模5万（k）小时的伪标记训练集的实验结果表明，基于TT的ST不仅显着减少了推理时间，而且还优于非流式级联ST进行英语 - 德语翻译。

自动副标题是将视听产品的语音自动转化为短文本的任务，换句话说，字幕及其相应的时间戳。生成的字幕需要符合多个空间和时间要求（长度，阅读速度），同时与语音同步并以促进理解的方式进行分割。鉴于其相当大的复杂性，迄今为止，通过分别处理转录，翻译，分割为字幕并预测时间戳的元素来解决自动字幕。在本文中，我们提出了第一个直接自动字幕模型，该模型在单个解决方案中从源语音中生成目标语言字幕及其时间戳。与经过内外数据和外域数据训练的最先进的级联模型的比较表明，我们的系统提供了高质量的字幕，同时在整合性方面也具有竞争力，并具有维护单个模型的所有优势。

同时语音转换（Simulst）是必须在部分，增量语音输入上执行输出生成的任务。近年来，由于交叉语言应用场景的传播，如国际现场会议和流媒体讲座，Sumulst已经变得很受欢迎，因为在飞行的语音翻译中可以促进用户访问视听内容。在本文中，我们分析到目前为止所开发的Simulst系统的特征，讨论其优势和缺点。然后我们专注于正确评估系统效率所需的评估框架。为此，我们提高了更广泛的性能分析的需求，还包括用户体验的角度。实际上，Simulst Systems不仅应在质量/延迟措施方面进行评估，而且还可以通过以任务为导向的指标计费，例如，用于所采用的可视化策略。鉴于此，我们突出了社区实现的目标以及仍然缺少的目标。

在本文中，我们介绍了一个高质量的大规模基准数据集，用于英语 - 越南语音翻译，其中有508音频小时，由331k的三胞胎组成（句子长度的音频，英语源笔录句，越南人目标subtitle句子）。我们还使用强基础进行了经验实验，发现传统的“级联”方法仍然优于现代“端到端”方法。据我们所知，这是第一个大规模的英语 - 越南语音翻译研究。我们希望我们的公开数据集和研究都可以作为未来研究和英语语音翻译应用的起点。我们的数据集可从https://github.com/vinairesearch/phost获得

End-to-end Speech Translation (E2E ST) aims to translate source speech into target translation without generating the intermediate transcript. However, existing approaches for E2E ST degrade considerably when only limited ST data are available. We observe that an ST model's performance strongly correlates with its embedding similarity from speech and transcript. In this paper, we propose Word-Aligned COntrastive learning (WACO), a novel method for few-shot speech-to-text translation. Our key idea is bridging word-level representations for both modalities via contrastive learning. We evaluate WACO and other methods on the MuST-C dataset, a widely used ST benchmark. Our experiments demonstrate that WACO outperforms the best baseline methods by 0.7-8.5 BLEU points with only 1-hour parallel data. Code is available at https://anonymous.4open.science/r/WACO .

While speech recognition Word Error Rate (WER) has reached human parity for English, long-form dictation scenarios still suffer from segmentation and punctuation problems resulting from irregular pausing patterns or slow speakers. Transformer sequence tagging models are effective at capturing long bi-directional context, which is crucial for automatic punctuation. Automatic Speech Recognition (ASR) production systems, however, are constrained by real-time requirements, making it hard to incorporate the right context when making punctuation decisions. In this paper, we propose a streaming approach for punctuation or re-punctuation of ASR output using dynamic decoding windows and measure its impact on punctuation and segmentation accuracy across scenarios. The new system tackles over-segmentation issues, improving segmentation F0.5-score by 13.9%. Streaming punctuation achieves an average BLEU-score improvement of 0.66 for the downstream task of Machine Translation (MT).

同时翻译系统在处理输入流中的部分源句子时开始产生输出。这些系统需要决定何时读取更多输入以及何时编写输出。这些决定取决于源/目标语言的结构以及部分输入序列中包含的信息。因此，读/写决策策略在不同的输入方式（即语音和文本）中保持不变。这激发了我们利用与语音输入相对应的文本成绩单，以改善同时的语音到文本翻译（Simulst）。我们建议通过同时使用文本到文本翻译（SIMULMT）任务来改善Simulst系统的决策政策，以改善Simulst系统的决策政策。我们还将几种技术从离线语音翻译域扩展，以探索Simulmt任务在改善Simulst性能中的作用。总体而言，我们在不同的延迟制度（ENDE）SIMULST任务的不同延迟制度中取得了34.66％ / 4.5 BLEU的改进。

由于其误差传播，延迟较少和更少的参数较少的潜力，端到端语音到文本翻译〜（e2e-st）变得越来越受欢迎。鉴于三联培训语料库$ \ langle演讲，转录，翻译\ rangle $，传统的高质量E2E-ST系统利用$ \ langle演讲，转录\ rangle $配对预先培训模型，然后利用$ \ Langle演讲，翻译\ rangle $配对进一步优化它。然而，该过程仅涉及每个阶段的两个元组数据，并且该松散耦合不能完全利用三重态数据之间的关联。在本文中，我们试图基于语音输入模拟转录和翻译的联合概率，以直接利用这种三重态数据。基于此，我们提出了一种新的正规化方法，用于改进三重态数据中双路分解协议的模型培训，理论上应该是相等的。为实现这一目标，我们将两个Kullback-Leibler发散正规化术语介绍到模型培训目的中，以减少双路径输出概率之间的不匹配。然后，训练有素的模型可以通过预定义的早期停止标签自然地被视为E2E-ST模型。 Must-C基准测试的实验表明，我们所提出的方法在所有8个语言对上显着优于最先进的E2E-ST基线，同时在自动语音识别任务中实现更好的性能。我们的代码在https://github.com/duyichao/e2e -st-tda开放。

语音细分将长言语分为短段，对于语音翻译（ST）至关重要。像WebRTC VAD这样的流行VAD工具通常依赖于基于暂停的细分。不幸的是，语音中的暂停不一定与句子边界匹配，句子可以通过很短的停顿连接，而VAD很难检测到。在这项研究中，我们建议使用使用分割的双语语音语料库训练的二元分类模型进行语音分割方法。我们还提出了一种结合VAD和上述语音分割方法的混合方法。实验结果表明，所提出的方法比常规分割方法更适合级联和端到端ST系统。混合方法进一步改善了翻译性能。

连接派时间分类（CTC）的模型在自动语音识别（ASR）方面具有吸引力，因为它们的非自动性性质。为了利用仅文本数据，语言模型（LM）集成方法（例如重新纠正和浅融合）已被广泛用于CTC。但是，由于需要降低推理速度，因此他们失去了CTC的非自动性性本质。在这项研究中，我们提出了一种使用电话条件的蒙版LM（PC-MLM）的误差校正方法。在提出的方法中，掩盖了来自CTC的贪婪解码输出中的较不自信的单词令牌。然后，PC-MLM预测这些蒙版的单词令牌给定的单词和手机补充了CTC。我们进一步将其扩展到可删除的PC-MLM，以解决插入错误。由于CTC和PC-MLM均为非自动回旋模型，因此该方法可以快速LM集成。在域适应设置中对自发日本（CSJ）和TED-LIUM2语料库进行的实验评估表明，我们所提出的方法在推理速度方面优于重新逆转和浅融合，并且在CSJ上的识别准确性方面。

最近的神经机翻译研究探索了灵活的发行订单，作为左右一代的替代品。然而，培训非单调模型带来了新的并发症：如何在同一最终结果到达的订单组合爆炸时搜索良好的订单？此外，这些自动排序如何与人类翻译的实际行为进行比较？目前的模型依靠手动构建的偏见或留下自己的所有可能性。在本文中，我们分析了人工后编辑所产生的排序，并使用它们培训自动编辑后系统。我们将生成的系统与由左右和随机编辑排序训练的人进行比较。我们观察到人类倾向于遵循几乎左右的顺序，而是有趣的偏差，例如首选通过纠正标点符号或动词而开始。

Speech translation (ST) is the task of directly translating acoustic speech signals in a source language into text in a foreign language. ST task has been addressed, for a long time, using a pipeline approach with two modules : first an Automatic Speech Recognition (ASR) in the source language followed by a text-to-text Machine translation (MT). In the past few years, we have seen a paradigm shift towards the end-to-end approaches using sequence-to-sequence deep neural network models. This paper presents our efforts towards the development of the first Broadcast News end-to-end Arabic to English speech translation system. Starting from independent ASR and MT LDC releases, we were able to identify about 92 hours of Arabic audio recordings for which the manual transcription was also translated into English at the segment level. These data was used to train and compare pipeline and end-to-end speech translation systems under multiple scenarios including transfer learning and data augmentation techniques.