How to solve the data scarcity problem for end-to-end speech-to-text translation (ST)? It's well known that data augmentation is an efficient method to improve performance for many tasks by enlarging the dataset. In this paper, we propose Mix at three levels for Speech Translation (M^3ST) method to increase the diversity of the augmented training corpus. Specifically, we conduct two phases of fine-tuning based on a pre-trained model using external machine translation (MT) data. In the first stage of fine-tuning, we mix the training corpus at three levels, including word level, sentence level and frame level, and fine-tune the entire model with mixed data. At the second stage of fine-tuning, we take both original speech sequences and original text sequences in parallel into the model to fine-tune the network, and use Jensen-Shannon divergence to regularize their outputs. Experiments on MuST-C speech translation benchmark and analysis show that M^3ST outperforms current strong baselines and achieves state-of-the-art results on eight directions with an average BLEU of 29.9.

Persona-based dialogue systems aim to generate consistent responses based on historical context and predefined persona. Unlike conventional dialogue generation, the persona-based dialogue needs to consider both dialogue context and persona, posing a challenge for coherent training. Specifically, this requires a delicate weight balance between context and persona. To achieve that, in this paper, we propose an effective framework with Persona-Adaptive Attention (PAA), which adaptively integrates the weights from the persona and context information via our designed attention. In addition, a dynamic masking mechanism is applied to the PAA to not only drop redundant information in context and persona but also serve as a regularization mechanism to avoid overfitting. Experimental results demonstrate the superiority of the proposed PAA framework compared to the strong baselines in both automatic and human evaluation. Moreover, the proposed PAA approach can perform equivalently well in a low-resource regime compared to models trained in a full-data setting, which achieve a similar result with only 20% to 30% of data compared to the larger models trained in the full-data setting. To fully exploit the effectiveness of our design, we designed several variants for handling the weighted information in different ways, showing the necessity and sufficiency of our weighting and masking designs.

在人类的言论中，说话者的态度不能只用文本内容完全表达。它必须带有语调。声明性的问题通常用于日常的广东话对话中，通常会以不断增长的语调发出。香草神经文本到语音（TTS）系统由于语义信息的丢失而无法为这些句子综合这些句子的上升。尽管使用额外的语言模型补充系统已经变得越来越普遍，但它们在建模升起的语调方面的性能尚未得到很好的研究。在本文中，我们建议通过基于BERT的语句/问题分类器来补充广州TTS模型。我们设计了不同的培训策略并比较他们的表现。我们对一个名为Cantts的粤语语料库进行实验。经验结果表明，单独的培训方法获得了最佳的概括性能和可行性。

本文研究了一种新型的预训练技术，该技术具有未配对的语音数据Segend2C，用于基于编码器的自动语音识别（ASR）。在一个多任务学习框架内，我们使用声音单元（即伪代码）介绍了编码器 - 编码器网络的两个预训练任务，这些任务来自离线聚类模型。一种是通过在编码器输出中通过掩盖语言建模来预测伪代码，例如Hubert模型，而另一个使解码器学会学会重建伪代码自动加工，而不是生成文本脚本。通过这种方式，解码器学会了在学习生成正确的文本之前先用代码重建原始语音信息。在Librispeech语料库上进行的综合实验表明，在没有解码器预训练的情况下，提出的Speek2C可以相对将单词错误率（WER）降低19.2％，并且在最先进的WAV2VEC 2.0和HUBERT上的表现显着优于微调子集为10h和100h。我们在https://github.com/microsoft/speecht5/tree/main/main/speech2c上发布代码和模型。

自我监督的语音表示学习在各种语音处理任务中显示出令人鼓舞的结果。但是，预先训练的模型，例如休伯特是存储密集型变压器，限制了其在低资源设置下的应用程序范围。为此，我们建议通过修剪结构化参数自动找到所需的体系结构Lighthubert，这是一个曾经是变压器的压缩框架。更确切地说，我们创建了一个基于变压器的超级网，该超网嵌套着数千个重量共享子网，并设计了一个两阶段的蒸馏策略，以利用休伯特的上下文化潜在表示。关于自动语音识别（ASR）和出色基准的实验表明，拟议的lighthubert可实现$ 10^9 $的架构，该体系结构涉及嵌入尺寸，注意力维度，头部编号，进率向前网络比率和网络深度。 Lighthubert优于ASR上的原始Hubert和Hubert大小的五个出色的任务，在大多数任务中，在大多数任务中都具有可比的性能，并减少了29％的参数，并获得了$ 3.5 \ times $ times $ compression $压缩比在三个超级任务中，例如自动扬声器验证，关键字发现和意图分类，略有准确的损失。代码和预培训模型可在https://github.com/mechanicalsea/lighthubert上找到。

自动音频标题（AAC）是一种跨模型翻译任务，旨在使用自然语言来描述音频剪辑的内容。如在DCEAD 2021挑战的任务6所接收的提交所示，这一问题已受到越来越兴趣的社区。现有的AAC系统通常基于编码器解码器架构，其中音频信号被编码为潜像表示，并与其对应的文本描述对齐，则使用解码器来生成标题。然而，AAC系统的培训经常遇到数据稀缺问题，这可能导致不准确的表示和音频文本对齐。为了解决这个问题，我们提出了一种名为对比损耗的新型编码器解码器框架（CL4AC）。在CL4AC中，通过对比样本来利用来自原始音频文本成对数据的自我监督信号来利用音频和文本之间的对应关系，该样本可以提高潜在表示的质量和音频和文本之间的对齐，同时训练有限的数据。实验是在披丁数据集上进行的，以显示我们提出的方法的有效性。

A Digital Twin (DT) is a simulation of a physical system that provides information to make decisions that add economic, social or commercial value. The behaviour of a physical system changes over time, a DT must therefore be continually updated with data from the physical systems to reflect its changing behaviour. For resource-constrained systems, updating a DT is non-trivial because of challenges such as on-board learning and the off-board data transfer. This paper presents a framework for updating data-driven DTs of resource-constrained systems geared towards system health monitoring. The proposed solution consists of: (1) an on-board system running a light-weight DT allowing the prioritisation and parsimonious transfer of data generated by the physical system; and (2) off-board robust updating of the DT and detection of anomalous behaviours. Two case studies are considered using a production gas turbine engine system to demonstrate the digital representation accuracy for real-world, time-varying physical systems.

Learning to predict masked tokens in a sequence has been shown to be a powerful pretraining objective for large-scale language models. After training, such masked language models can provide distributions of tokens conditioned on bidirectional context. In this short draft, we show that such bidirectional conditionals often demonstrate considerable inconsistencies, i.e., they can not be derived from a coherent joint distribution when considered together. We empirically quantify such inconsistencies in the simple scenario of bigrams for two common styles of masked language models: T5-style and BERT-style. For example, we show that T5 models often confuse its own preference regarding two similar bigrams. Such inconsistencies may represent a theoretical pitfall for the research work on sampling sequences based on the bidirectional conditionals learned by BERT-style MLMs. This phenomenon also means that T5-style MLMs capable of infilling will generate discrepant results depending on how much masking is given, which may represent a particular trust issue.

We study the problem of planning under model uncertainty in an online meta-reinforcement learning (RL) setting where an agent is presented with a sequence of related tasks with limited interactions per task. The agent can use its experience in each task and across tasks to estimate both the transition model and the distribution over tasks. We propose an algorithm to meta-learn the underlying structure across tasks, utilize it to plan in each task, and upper-bound the regret of the planning loss. Our bound suggests that the average regret over tasks decreases as the number of tasks increases and as the tasks are more similar. In the classical single-task setting, it is known that the planning horizon should depend on the estimated model's accuracy, that is, on the number of samples within task. We generalize this finding to meta-RL and study this dependence of planning horizons on the number of tasks. Based on our theoretical findings, we derive heuristics for selecting slowly increasing discount factors, and we validate its significance empirically.

The current reinforcement learning algorithm uses forward-generated trajectories to train the agent. The forward-generated trajectories give the agent little guidance, so the agent can explore as much as possible. While the appreciation of reinforcement learning comes from enough exploration, this gives the trade-off of losing sample efficiency. The sampling efficiency is an important factor that decides the performance of the algorithm. Past tasks use reward shaping techniques and changing the structure of the network to increase sample efficiency, however these methods require many steps to implement. In this work, we propose novel reverse curriculum reinforcement learning. Reverse curriculum learning starts training the agent using the backward trajectory of the episode rather than the original forward trajectory. This gives the agent a strong reward signal, so the agent can learn in a more sample-efficient manner. Moreover, our method only requires a minor change in algorithm, which is reversing the order of trajectory before training the agent. Therefore, it can be simply applied to any state-of-art algorithms.