In recent years, the development of accurate deep keyword spotting (KWS) models has resulted in KWS technology being embedded in a number of technologies such as voice assistants. Many of these models rely on large amounts of labelled data to achieve good performance. As a result, their use is restricted to applications for which a large labelled speech data set can be obtained. Self-supervised learning seeks to mitigate the need for large labelled data sets by leveraging unlabelled data, which is easier to obtain in large amounts. However, most self-supervised methods have only been investigated for very large models, whereas KWS models are desired to be small. In this paper, we investigate the use of self-supervised pretraining for the smaller KWS models in a label-deficient scenario. We pretrain the Keyword Transformer model using the self-supervised framework Data2Vec and carry out experiments on a label-deficient setup of the Google Speech Commands data set. It is found that the pretrained models greatly outperform the models without pretraining, showing that Data2Vec pretraining can increase the performance of KWS models in label-deficient scenarios. The source code is made publicly available.

我们总结了使用巨大的自动语音识别（ASR）模型的大量努力的结果，该模型使用包含大约一百万小时音频的大型，多样的未标记数据集进行了预训练。我们发现，即使对于拥有数万个小时的标记数据的非常大的任务，预训练，自我培训和扩大模型大小的组合也大大提高了数据效率。特别是，在具有34K小时标记数据的ASR任务上，通过微调80亿个参数预先训练的构象异构体模型，我们可以匹配最先进的（SOTA）性能（SOTA）的性能，只有3％的培训数据和通过完整的训练集可以显着改善SOTA。我们还报告了从使用大型预训练和自我训练的模型来完成一系列下游任务所获得的普遍利益，这些任务涵盖了广泛的语音域，并涵盖了多个数据集大小的大小，包括在许多人中获得SOTA性能公共基准。此外，我们利用预先训练的网络的学会表示，在非ASR任务上实现SOTA结果。

Current self-supervised learning algorithms are often modality-specific and require large amounts of computational resources. To address these issues, we increase the training efficiency of data2vec, a learning objective that generalizes across several modalities. We do not encode masked tokens, use a fast convolutional decoder and amortize the effort to build teacher representations. data2vec 2.0 benefits from the rich contextualized target representations introduced in data2vec which enable a fast self-supervised learner. Experiments on ImageNet-1K image classification show that data2vec 2.0 matches the accuracy of Masked Autoencoders in 16.4x lower pre-training time, on Librispeech speech recognition it performs as well as wav2vec 2.0 in 10.6x less time, and on GLUE natural language understanding it matches a retrained RoBERTa model in half the time. Trading some speed for accuracy results in ImageNet-1K top-1 accuracy of 86.8\% with a ViT-L model trained for 150 epochs.

We present RAVEn, a self-supervised multi-modal approach to jointly learn visual and auditory speech representations. Our pre-training objective involves encoding masked inputs, and then predicting contextualised targets generated by slowly-evolving momentum encoders. Driven by the inherent differences between video and audio, our design is asymmetric w.r.t. the two modalities' pretext tasks: Whereas the auditory stream predicts both the visual and auditory targets, the visual one predicts only the auditory targets. We observe strong results in low- and high-resource labelled data settings when fine-tuning the visual and auditory encoders resulting from a single pre-training stage, in which the encoders are jointly trained. Notably, RAVEn surpasses all self-supervised methods on visual speech recognition (VSR) on LRS3, and combining RAVEn with self-training using only 30 hours of labelled data even outperforms a recent semi-supervised method trained on 90,000 hours of non-public data. At the same time, we achieve state-of-the-art results in the LRS3 low-resource setting for auditory speech recognition (as well as for VSR). Our findings point to the viability of learning powerful speech representations entirely from raw video and audio, i.e., without relying on handcrafted features. Code and models will be made public.

Self-supervised approaches for speech representation learning are challenged by three unique problems: (1) there are multiple sound units in each input utterance, (2) there is no lexicon of input sound units during the pre-training phase, and (3) sound units have variable lengths with no explicit segmentation. To deal with these three problems, we propose the Hidden-Unit BERT (HuBERT) approach for self-supervised speech representation learning, which utilizes an offline clustering step to provide aligned target labels for a BERT-like prediction loss. A key ingredient of our approach is applying the prediction loss over the masked regions only, which forces the model to learn a combined acoustic and language model over the continuous inputs. HuBERT relies primarily on the consistency of the unsupervised clustering step rather than the intrinsic quality of the assigned cluster labels. Starting with a simple k-means teacher of 100 clusters, and using two iterations of clustering, the HuBERT model either matches or improves upon the state-ofthe-art wav2vec 2.0 performance on the Librispeech (960h) and Libri-light (60,000h) benchmarks with 10min, 1h, 10h, 100h, and 960h fine-tuning subsets. Using a 1B parameter model, HuBERT shows up to 19% and 13% relative WER reduction on the more challenging dev-other and test-other evaluation subsets. 1

The massive growth of self-supervised learning (SSL) has been witnessed in language, vision, speech, and audio domains over the past few years. While discrete label prediction is widely adopted for other modalities, the state-of-the-art audio SSL models still employ reconstruction loss for pre-training. Compared with reconstruction loss, semantic-rich discrete label prediction encourages the SSL model to abstract the high-level audio semantics and discard the redundant details as in human perception. However, a semantic-rich acoustic tokenizer for general audio pre-training is usually not straightforward to obtain, due to the continuous property of audio and unavailable phoneme sequences like speech. To tackle this challenge, we propose BEATs, an iterative audio pre-training framework to learn Bidirectional Encoder representation from Audio Transformers, where an acoustic tokenizer and an audio SSL model are optimized by iterations. In the first iteration, we use random projection as the acoustic tokenizer to train an audio SSL model in a mask and label prediction manner. Then, we train an acoustic tokenizer for the next iteration by distilling the semantic knowledge from the pre-trained or fine-tuned audio SSL model. The iteration is repeated with the hope of mutual promotion of the acoustic tokenizer and audio SSL model. The experimental results demonstrate our acoustic tokenizers can generate discrete labels with rich audio semantics and our audio SSL models achieve state-of-the-art results across various audio classification benchmarks, even outperforming previous models that use more training data and model parameters significantly. Specifically, we set a new state-of-the-art mAP 50.6% on AudioSet-2M for audio-only models without using any external data, and 98.1% accuracy on ESC-50. The code and pre-trained models are available at https://aka.ms/beats.

这项工作介绍了Brillsson，这是一种基于二进制神经网络的新型表示模型，用于广泛的非语义语音任务。我们从一个大型且价值的琐事模型中使用知识蒸馏来训练该模型，其中仅用于训练Trillsson的数据集中只有一小部分。由此产生的Brillsson型号的尺寸仅为2MB，潜伏期小于8ms，使其适合在低资源设备（例如可穿戴设备）中部署。我们在八项基准任务（包括但不限于口语识别，情感识别，荒地状况诊断和关键字斑点）上评估布里尔森，并证明我们提出的拟议的超轻质和低延迟模型以及大型模型以及大型模型。

最近，蒙面的预测预训练在自我监督的学习（SSL）方面取得了显着的进展，以进行语音识别。它通常需要以无监督的方式获得的代码簿，从而使其准确和难以解释。我们提出了两种监督指导的代码书生成方法，以提高自动语音识别（ASR）的性能以及预训练效率，要么通过使用混合ASR系统来解码以生成音素级别对准（命名为PBERT），要么通过在上进行集群进行聚类。从端到端CTC模型（命名CTC聚类）提取的监督语音功能。混合动力和CTC模型均经过与微调相同的少量标记语音训练。实验表明，我们的方法对各种SSL和自我训练基准的优势具有显着优势，相对减少了17.0％。我们的预训练模型在非ASR语音任务中还显示出良好的可传递性。

变压器已经看到了自然语言处理和计算机视觉任务的前所未有的上升。但是，在音频任务中，由于音频波形的极大序列长度或在培训基于傅立叶特征时，它们是不可行的。在这项工作中，我们介绍了一个架构，Audiomer，在那里我们将1D残差网络与表演者的注意力结合起来，以实现使用原始音频波形的关键字在关键字中实现最先进的性能，优先于以前的所有方法，同时计算更便宜和参数效率。此外，我们的模型具有语音处理的实际优点，例如由于缺乏位置编码而在任意长的音频剪辑上推断。代码可在https://github.com/the-learning-machines/dautiomer获得

知识蒸馏（KD），最称为模型压缩的有效方法，旨在将更大的网络（教师）的知识转移到更小的网络（学生）。传统的KD方法通常采用以监督方式培训的教师模型，其中输出标签仅作为目标处理。我们进一步扩展了这一受监督方案，我们为KD，即Oracle老师推出了一种新型的教师模型，它利用源输入和输出标签的嵌入来提取更准确的知识来转移到学生。所提出的模型遵循变压器网络的编码器解码器注意结构，这允许模型从输出标签上参加相关信息。在三种不同的序列学习任务中进行了广泛的实验：语音识别，场景文本识别和机器翻译。从实验结果来看，我们经验证明，拟议的模型在这些任务中改善了学生，同时在教师模型的培训时间内实现了相当大的速度。

我们介绍BERTPHONE，一个在大型语音上培训的变压器编码器，输出可以用于扬声器和语言识别的语音感知的上下文表示向量。这是通过对两个目标的培训来实现的：首先是通过调整伯特对连续领域的启发，涉及掩蔽输入框架的跨度并重建用于声学表示学习的整个序列;其次，由ASR的瓶颈特征成功的启发是应用于音素标签的序列级CTC损失，用于语音表示学习。我们预留了两种BERTPHONE型号（一个在FISHER上，一个在TED-lium上），并用它们用作两个任务的X-Vector-Sique DNN中的特征提取器。我们达到最先进的$ C _ {\ TEXT {AVG}} $ 6.16就具有挑战性的LRE07 3SEC封闭式语言识别任务。在Fisher和VoxceleB扬声器识别任务上，我们在培训BertPhone向量而不是MFCC时，我们看到扬声器EER的相对减少18％。通常，BERTPHONE在同一数据上优于先前的语音预制方法。我们在https://github.com/awslabs/speech -representations释放我们的代码和模型。

受到计算机视觉的自我监督学习的最新进展的启发，在本文中，我们介绍了Delores，这是一种新的通用音频表示方法。我们的主要目标是使我们的网络学习在资源受限的设置（数据和计算）中，可以很好地跨越各种下游任务。受Barlow Twins目标功能的启发，我们建议学习对输入音频样本失真不变的嵌入，同时确保它们包含有关样本的非冗余信息。为此，我们测量了两个相同的网络的输出之间的互相关矩阵，该网络用从音频文件采样的音频段的变形版本中，使其尽可能接近身份矩阵。我们将大规模音频集数据集和FSD50K的一小部分组合用于自学学习，并且与最先进的算法相比，参数的一半不到一半。为了进行评估，我们将这些学习的表示形式转移到9个下游分类任务，包括语音，音乐和动物声音，并在不同的评估设置下显示竞争结果。除了简单明了，我们的预训练算法还可以通过其固有的构造本质来计算，并且不需要仔细的实施细节以避免琐碎或退化的解决方案。此外，我们对结果进行消融研究，并使我们的所有代码和预培训模型公开可用https://github.com/speech-lab-iitm/delores。

自我监督的语音表示学习在各种语音处理任务中显示出令人鼓舞的结果。但是，预先训练的模型，例如休伯特是存储密集型变压器，限制了其在低资源设置下的应用程序范围。为此，我们建议通过修剪结构化参数自动找到所需的体系结构Lighthubert，这是一个曾经是变压器的压缩框架。更确切地说，我们创建了一个基于变压器的超级网，该超网嵌套着数千个重量共享子网，并设计了一个两阶段的蒸馏策略，以利用休伯特的上下文化潜在表示。关于自动语音识别（ASR）和出色基准的实验表明，拟议的lighthubert可实现$ 10^9 $的架构，该体系结构涉及嵌入尺寸，注意力维度，头部编号，进率向前网络比率和网络深度。 Lighthubert优于ASR上的原始Hubert和Hubert大小的五个出色的任务，在大多数任务中，在大多数任务中都具有可比的性能，并减少了29％的参数，并获得了$ 3.5 \ times $ times $ compression $压缩比在三个超级任务中，例如自动扬声器验证，关键字发现和意图分类，略有准确的损失。代码和预培训模型可在https://github.com/mechanicalsea/lighthubert上找到。

自动图像分类是食品科学中监督机器学习的常见任务。一个例子是基于图像的水果外部质量或成熟度的分类。为此，通常使用深层卷积神经网络（CNN）。这些模型通常需要大量标记的培训样本和增强的计算资源。尽管商业水果分类线很容易满足这些要求，但这些先决条件可能会阻碍机器学习方法的使用，尤其是对于发展中国家的小农户。我们提出了一种基于预先训练的视觉变压器（VIT）的替代方法，该方法特别适用于数据可用性较低和计算资源有限的域。可以在标准设备上使用有限的资源来轻松实施，这可以使这些模型在发展中国家的基于智能手机的图像分类中民主化。我们通过用良好的CNN方法基准对香蕉和苹果水果的域数据集进行两项不同的分类任务来证明我们方法的竞争力。我们的方法在3745张图像的训练数据集上，分类精度低于表现最佳的CNN（0.950 vs. 0.958）的分类精度。同时，当只有少量标记的训练样本可用时，我们的方法是优越的。与CNN相比，它需要少三倍才能达到0.90的精度。此外，低维特征嵌入的可视化表明，我们的研究中使用的模型从看不见的数据中提取了出色的特征，而无需分配标签。

Self-supervised learning via masked prediction pre-training (MPPT) has shown impressive performance on a range of speech-processing tasks. This paper proposes a method to bias self-supervised learning towards a specific task. The core idea is to slightly finetune the model that is used to obtain the target sequence. This leads to better performance and a substantial increase in training speed. Furthermore, this paper proposes a variant of MPPT that allows low-footprint streaming models to be trained effectively by computing the MPPT loss on masked and unmasked frames. These approaches are evaluated for automatic speech recognition on the Librispeech corpus, where 100 hours of data served as the labelled data and 860 hours as the unlabelled data. The biased training outperforms the unbiased training by 15.5% after 250k updates and 23.8% after 100k updates on test-other. For the streaming models, the pre-training approach yields a reduction in word error rate of 44.1%.

One paradigm for learning from few labeled examples while making best use of a large amount of unlabeled data is unsupervised pretraining followed by supervised fine-tuning. Although this paradigm uses unlabeled data in a task-agnostic way, in contrast to common approaches to semi-supervised learning for computer vision, we show that it is surprisingly effective for semi-supervised learning on ImageNet. A key ingredient of our approach is the use of big (deep and wide) networks during pretraining and fine-tuning. We find that, the fewer the labels, the more this approach (task-agnostic use of unlabeled data) benefits from a bigger network. After fine-tuning, the big network can be further improved and distilled into a much smaller one with little loss in classification accuracy by using the unlabeled examples for a second time, but in a task-specific way. The proposed semi-supervised learning algorithm can be summarized in three steps: unsupervised pretraining of a big ResNet model using SimCLRv2, supervised fine-tuning on a few labeled examples, and distillation with unlabeled examples for refining and transferring the task-specific knowledge. This procedure achieves 73.9% ImageNet top-1 accuracy with just 1% of the labels (≤13 labeled images per class) using ResNet-50, a 10× improvement in label efficiency over the previous state-of-theart. With 10% of labels, ResNet-50 trained with our method achieves 77.5% top-1 accuracy, outperforming standard supervised training with all of the labels. 1

最先进的自动语音识别（ASR）系统经过数以万计的标记语音数据训练。人类转录很昂贵且耗时。诸如转录的质量和一致性之类的因素可以极大地影响使用这些数据训练的ASR模型的性能。在本文中，我们表明我们可以通过利用最近的自学和半监督学习技术来培训强大的教师模型来生产高质量的伪标签。具体来说，我们仅使用（无监督/监督培训）和迭代嘈杂的学生教师培训来培训6亿个参数双向教师模型。该模型在语音搜索任务上达到了4.0％的单词错误率（WER），比基线相对好11.1％。我们进一步表明，通过使用这种强大的教师模型来生成用于训练的高质量伪标签，与使用人类标签相比，流媒体模型可以实现13.6％的相对减少（5.9％至5.1％）。

随着虚拟助手变得越来越多样化和专业，对应用或特定品牌唤醒的需求也是如此。但是，通常用于训练尾流检测器的特定于唤醒特定的数据集是昂贵的。在本文中，我们探索了两种技术来利用声音建模数据，以提高大唱歌的语音识别，以改善专用的尾流探测器：转移学习和知识蒸馏。我们还探讨了这些技术如何与时间同步训练目标相互作用以提高检测潜伏期。实验显示在开源“嘿STHIPS”数据集中，并且内部远场数据集更具挑战性。使用大型声学模型中的电话同步目标和知识蒸馏，我们能够提高两个数据集的数据集尺寸的精度，同时降低延迟。

神经网络可以从单个图像中了解视觉世界的内容是什么？虽然它显然不能包含存在的可能对象，场景和照明条件 - 在所有可能的256 ^（3x224x224）224尺寸的方形图像中，它仍然可以在自然图像之前提供强大的。为了分析这一假设，我们通过通过监控掠夺教师的知识蒸馏来制定一种训练神经网络的培训神经网络。有了这个，我们发现上述问题的答案是：“令人惊讶的是，很多”。在定量术语中，我们在CiFar-10/100上找到了94％/ 74％的前1个精度，在想象中，通过将这种方法扩展到音频，84％的语音组合。在广泛的分析中，我们解除了增强，源图像和网络架构的选择，以及在从未见过熊猫的网络中发现“熊猫神经元”。这项工作表明，一个图像可用于推断成千上万的对象类，并激励关于增强和图像的基本相互作用的更新的研究议程。