Convolutional Neural Networks (CNNs) have proven very effective in image classification and show promise for audio. We use various CNN architectures to classify the soundtracks of a dataset of 70M training videos (5.24 million hours) with 30,871 video-level labels. We examine fully connected Deep Neural Networks (DNNs), AlexNet [1], VGG [2], Inception [3], and ResNet [4]. We investigate varying the size of both training set and label vocabulary, finding that analogs of the CNNs used in image classification do well on our audio classification task, and larger training and label sets help up to a point. A model using embeddings from these classifiers does much better than raw features on the Audio Set [5] Acoustic Event Detection (AED) classification task.

音频标记是一个活跃的研究区，具有广泛的应用。自发布以来，在推进模型性能方面取得了很大进展，主要来自新颖的模型架构和注意力模块。但是，我们发现适当的培训技术对于使用音频构建音频标记模型同样重要，但没有得到他们应得的关注。为了填补差距，在这项工作中，我们呈现PSLA，一系列培训技术，可以明显增强模型准确性，包括想象成预测，平衡采样，数据增强，标签增强，模型聚集和其设计选择。通过使用这些技术培训效率，我们可以分别获得单个型号（具有13.6M参数）和一个集合模型，分别实现Audioset的平均平均精度（MAP）分数为0.444和0.474，优于81米的先前最佳系统0.439参数。此外，我们的型号还在FSD50K上实现了0.567的新型地图。

We consider the question: what can be learnt by looking at and listening to a large number of unlabelled videos? There is a valuable, but so far untapped, source of information contained in the video itself -the correspondence between the visual and the audio streams, and we introduce a novel "Audio-Visual Correspondence" learning task that makes use of this. Training visual and audio networks from scratch, without any additional supervision other than the raw unconstrained videos themselves, is shown to successfully solve this task, and, more interestingly, result in good visual and audio representations. These features set the new state-of-the-art on two sound classification benchmarks, and perform on par with the state-of-the-art selfsupervised approaches on ImageNet classification. We also demonstrate that the network is able to localize objects in both modalities, as well as perform fine-grained recognition tasks.

Convolutional neural networks (CNNs) have been extensively applied for image recognition problems giving stateof-the-art results on recognition, detection, segmentation and retrieval. In this work we propose and evaluate several deep neural network architectures to combine image information across a video over longer time periods than previously attempted. We propose two methods capable of handling full length videos. The first method explores various convolutional temporal feature pooling architectures, examining the various design choices which need to be made when adapting a CNN for this task. The second proposed method explicitly models the video as an ordered sequence of frames. For this purpose we employ a recurrent neural network that uses Long Short-Term Memory (LSTM) cells which are connected to the output of the underlying CNN. Our best networks exhibit significant performance improvements over previously published results on the Sports 1 million dataset (73.1% vs. 60.9%) and the UCF-101 datasets with (88.6% vs. 88.0%) and without additional optical flow information (82.6% vs. 73.0%).

用于标记和分类声信号的标准机器学习模型无法处理训练过程中未见的类。通过基于适应性的类描述来预测类，零射击（ZS）学习克服了这一限制。这项研究旨在研究基于自我注意力的音频嵌入体系结构对ZS学习的有效性。为此，我们将最近的贴布频谱变压器与两个经典的卷积体系结构进行了比较。我们在三个任务和三个不同的基准数据集上评估了这三个架构：在Audioset上的通用标记，ESC-50上的环境声音分类以及OpenMIC上的仪器标记。我们的结果表明，基于自我注意的嵌入方法的表现都优于所有这些设置中的卷积架构。通过相应地设计培训和测试数据，我们观察到，当训练和新测试类之间的“语义距离”很大时，预测性能会大大受到影响，这种效果值得进行更详细的研究。

Convolutional Neural Networks (CNNs) have been established as a powerful class of models for image recognition problems. Encouraged by these results, we provide an extensive empirical evaluation of CNNs on largescale video classification using a new dataset of 1 million YouTube videos belonging to 487 classes. We study multiple approaches for extending the connectivity of a CNN in time domain to take advantage of local spatio-temporal information and suggest a multiresolution, foveated architecture as a promising way of speeding up the training. Our best spatio-temporal networks display significant performance improvements compared to strong feature-based baselines (55.3% to 63.9%), but only a surprisingly modest improvement compared to single-frame models (59.3% to 60.9%). We further study the generalization performance of our best model by retraining the top layers on the UCF-101 Action Recognition dataset and observe significant performance improvements compared to the UCF-101 baseline model (63.3% up from 43.9%).

Deep neural networks (DNNs) are currently widely used for many artificial intelligence (AI) applications including computer vision, speech recognition, and robotics. While DNNs deliver state-of-the-art accuracy on many AI tasks, it comes at the cost of high computational complexity. Accordingly, techniques that enable efficient processing of DNNs to improve energy efficiency and throughput without sacrificing application accuracy or increasing hardware cost are critical to the wide deployment of DNNs in AI systems.This article aims to provide a comprehensive tutorial and survey about the recent advances towards the goal of enabling efficient processing of DNNs. Specifically, it will provide an overview of DNNs, discuss various hardware platforms and architectures that support DNNs, and highlight key trends in reducing the computation cost of DNNs either solely via hardware design changes or via joint hardware design and DNN algorithm changes. It will also summarize various development resources that enable researchers and practitioners to quickly get started in this field, and highlight important benchmarking metrics and design considerations that should be used for evaluating the rapidly growing number of DNN hardware designs, optionally including algorithmic co-designs, being proposed in academia and industry.The reader will take away the following concepts from this article: understand the key design considerations for DNNs; be able to evaluate different DNN hardware implementations with benchmarks and comparison metrics; understand the trade-offs between various hardware architectures and platforms; be able to evaluate the utility of various DNN design techniques for efficient processing; and understand recent implementation trends and opportunities.

音频分割和声音事件检测是机器聆听中的关键主题，旨在检测声学类别及其各自的边界。它对于音频分析，语音识别，音频索引和音乐信息检索非常有用。近年来，大多数研究文章都采用分类。该技术将音频分为小帧，并在这些帧上单独执行分类。在本文中，我们提出了一种新颖的方法，叫您只听一次（Yoho），该方法受到计算机视觉中普遍采用的Yolo算法的启发。我们将声学边界的检测转换为回归问题，而不是基于框架的分类。这是通过具有单独的输出神经元来检测音频类的存在并预测其起点和终点来完成的。与最先进的卷积复发性神经网络相比，Yoho的F量的相对改善范围从多个数据集中的1％到6％不等，以进行音频分段和声音事件检测。由于Yoho的输出更端到端，并且可以预测的神经元更少，因此推理速度的速度至少比逐个分类快6倍。另外，由于这种方法可以直接预测声学边界，因此后处理和平滑速度约为7倍。

The paucity of videos in current action classification datasets (UCF-101 and HMDB-51) has made it difficult to identify good video architectures, as most methods obtain similar performance on existing small-scale benchmarks. This paper re-evaluates state-of-the-art architectures in light of the new Kinetics Human Action Video dataset. Kinetics has two orders of magnitude more data, with 400 human action classes and over 400 clips per class, and is collected from realistic, challenging YouTube videos. We provide an analysis on how current architectures fare on the task of action classification on this dataset and how much performance improves on the smaller benchmark datasets after pre-training on Kinetics.We also introduce a new Two-Stream Inflated 3D Con-vNet (I3D) that is based on 2D ConvNet inflation: filters and pooling kernels of very deep image classification ConvNets are expanded into 3D, making it possible to learn seamless spatio-temporal feature extractors from video while leveraging successful ImageNet architecture designs and even their parameters. We show that, after pre-training on Kinetics, I3D models considerably improve upon the state-of-the-art in action classification, reaching 80.9% on HMDB-51 and 98.0% on UCF-101.

Humans perceive the world by concurrently processing and fusing high-dimensional inputs from multiple modalities such as vision and audio. Machine perception models, in stark contrast, are typically modality-specific and optimised for unimodal benchmarks, and hence late-stage fusion of final representations or predictions from each modality (`late-fusion') is still a dominant paradigm for multimodal video classification. Instead, we introduce a novel transformer based architecture that uses `fusion bottlenecks' for modality fusion at multiple layers. Compared to traditional pairwise self-attention, our model forces information between different modalities to pass through a small number of bottleneck latents, requiring the model to collate and condense the most relevant information in each modality and only share what is necessary. We find that such a strategy improves fusion performance, at the same time reducing computational cost. We conduct thorough ablation studies, and achieve state-of-the-art results on multiple audio-visual classification benchmarks including Audioset, Epic-Kitchens and VGGSound. All code and models will be released.

Convolutional networks are at the core of most stateof-the-art computer vision solutions for a wide variety of tasks. Since 2014 very deep convolutional networks started to become mainstream, yielding substantial gains in various benchmarks. Although increased model size and computational cost tend to translate to immediate quality gains for most tasks (as long as enough labeled data is provided for training), computational efficiency and low parameter count are still enabling factors for various use cases such as mobile vision and big-data scenarios. Here we are exploring ways to scale up networks in ways that aim at utilizing the added computation as efficiently as possible by suitably factorized convolutions and aggressive regularization. We benchmark our methods on the ILSVRC 2012 classification challenge validation set demonstrate substantial gains over the state of the art: 21.2% top-1 and 5.6% top-5 error for single frame evaluation using a network with a computational cost of 5 billion multiply-adds per inference and with using less than 25 million parameters. With an ensemble of 4 models and multi-crop evaluation, we report 3.5% top-5 error and 17.3% top-1 error.

Transfer learning is a cornerstone of computer vision, yet little work has been done to evaluate the relationship between architecture and transfer. An implicit hypothesis in modern computer vision research is that models that perform better on ImageNet necessarily perform better on other vision tasks. However, this hypothesis has never been systematically tested. Here, we compare the performance of 16 classification networks on 12 image classification datasets. We find that, when networks are used as fixed feature extractors or fine-tuned, there is a strong correlation between ImageNet accuracy and transfer accuracy (r = 0.99 and 0.96, respectively). In the former setting, we find that this relationship is very sensitive to the way in which networks are trained on ImageNet; many common forms of regularization slightly improve ImageNet accuracy but yield penultimate layer features that are much worse for transfer learning. Additionally, we find that, on two small fine-grained image classification datasets, pretraining on ImageNet provides minimal benefits, indicating the learned features from Ima-geNet do not transfer well to fine-grained tasks. Together, our results show that ImageNet architectures generalize well across datasets, but ImageNet features are less general than previously suggested.

这项工作旨在自动评估儿童的语言发展是否适合年龄。经过验证的语音和语言测试用于此目的测试听觉记忆。在这项工作中，任务是确定是否正确说出了口语非单词。我们比较有动机来建模特定语言结构的不同方法：低水平特征（FFT），扬声器嵌入（ECAPA-TDNN），素化 - 动机的嵌入（WAV2VEC 2.0）和语音嵌入Senones（ASR ASR ACOSTIC模型）形式。每种方法都提供了类似VGG的5层CNN分类器的输入。我们还检查了每个非单词的适应性。使用来自口头非单词的不同幼儿园的录音进行了对拟议系统的评估。 ECAPA-TDNN和低级FFT特征不会明确模型语音信息； WAV2VEC2.0经过素数标签训练，我们的ASR声学模型包含（子）语音信息。我们发现，语音建模越颗粒状，达到的识别率就越高。在ASR声学模型特征上训练的最佳系统的精度为89.4％，在ROC（接收器操作特征）曲线（AUC）下的面积为0.923。与FFT-BASELINE相比，这对应于20.2％和AUC相对0.309的改善。

The purpose of this study is to determine whether current video datasets have sufficient data for training very deep convolutional neural networks (CNNs) with spatio-temporal three-dimensional (3D) kernels. Recently, the performance levels of 3D CNNs in the field of action recognition have improved significantly. However, to date, conventional research has only explored relatively shallow 3D architectures. We examine the architectures of various 3D CNNs from relatively shallow to very deep ones on current video datasets. Based on the results of those experiments, the following conclusions could be obtained: (i) training resulted in significant overfitting for UCF-101, HMDB-51, and Ac-tivityNet but not for Kinetics. (ii) The Kinetics dataset has sufficient data for training of deep 3D CNNs, and enables training of up to 152 ResNets layers, interestingly similar to 2D ResNets on ImageNet. ResNeXt-101 achieved 78.4% average accuracy on the Kinetics test set. (iii) Kinetics pretrained simple 3D architectures outperforms complex 2D architectures, and the pretrained ResNeXt-101 achieved 94.5% and 70.2% on respectively. The use of 2D CNNs trained on ImageNet has produced significant progress in various tasks in image. We believe that using deep 3D CNNs together with Kinetics will retrace the successful history of 2D CNNs and ImageNet, and stimulate advances in computer vision for videos. The codes and pretrained models used in this study are publicly available1.

We propose a general framework called Network Dissection for quantifying the interpretability of latent representations of CNNs by evaluating the alignment between individual hidden units and a set of semantic concepts. Given any CNN model, the proposed method draws on a broad data set of visual concepts to score the semantics of hidden units at each intermediate convolutional layer. The units with semantics are given labels across a range of objects, parts, scenes, textures, materials, and colors. We use the proposed method to test the hypothesis that interpretability of units is equivalent to random linear combinations of units, then we apply our method to compare the latent representations of various networks when trained to solve different supervised and self-supervised training tasks. We further analyze the effect of training iterations, compare networks trained with different initializations, examine the impact of network depth and width, and measure the effect of dropout and batch normalization on the interpretability of deep visual representations. We demonstrate that the proposed method can shed light on characteristics of CNN models and training methods that go beyond measurements of their discriminative power.

纯粹后的损害评估对于管理资源分配和执行有效响应至关重要。传统上，这种评估是通过野外侦察进行的，该侦察速度缓慢，危险且艰巨。取而代之的是，在本文中，我们进一步提出了通过卷积神经网络实施深度学习的想法，以便将建筑物的后卫星卫星图像分类为被洪水/损坏或未损坏的。该实验是在2017年哈维飓风之后使用的，该数据集采用了一个包含大休斯顿地区的纯种后卫星图像的数据集进行。本文实施了三个卷积神经网络模型体系结构，并配对了其他模型考虑，以实现高精度（超过99％），（超过99％），，超过99％），（超过99％）加强在殖民后灾难评估中有效使用机器学习。

In this study, we systematically investigate the impact of class imbalance on classification performance of convolutional neural networks (CNNs) and compare frequently used methods to address the issue. Class imbalance is a common problem that has been comprehensively studied in classical machine learning, yet very limited systematic research is available in the context of deep learning. In our study, we use three benchmark datasets of increasing complexity, MNIST, CIFAR-10 and ImageNet, to investigate the effects of imbalance on classification and perform an extensive comparison of several methods to address the issue: oversampling, undersampling, two-phase training, and thresholding that compensates for prior class probabilities. Our main evaluation metric is area under the receiver operating characteristic curve (ROC AUC) adjusted to multi-class tasks since overall accuracy metric is associated with notable difficulties in the context of imbalanced data. Based on results from our experiments we conclude that (i) the effect of class imbalance on classification performance is detrimental; (ii) the method of addressing class imbalance that emerged as dominant in almost all analyzed scenarios was oversampling; (iii) oversampling should be applied to the level that completely eliminates the imbalance, whereas the optimal undersampling ratio depends on the extent of imbalance; (iv) as opposed to some classical machine learning models, oversampling does not cause overfitting of CNNs; (v) thresholding should be applied to compensate for prior class probabilities when overall number of properly classified cases is of interest.

犯罪率与人口的增加率成比例地增加。最突出的方法是引入基于闭路电视（CCTV）相机的监视以解决问题。视频监控摄像机增加了一个新的维度来检测犯罪。目前正在进行自动安全摄像机监控的几项研究工作，基本目标是从视频饲料发现暴力活动。从技术方面来看，这是一个具有挑战性的问题，因为分析了一组帧，即时间维度的视频，以检测暴力可能需要仔细的机器学习模型训练，以减少错误的结果。本研究通过整合最先进的深度学习方法来重点介绍该问题，以确保用于检测暴力活动的自主监测的强大管道，例如，踢，冲压和拍打。最初，我们设计了这种特定兴趣的数据集，其中包含600个视频（每个动作200个）。稍后，我们已经利用现有的预先训练的模型架构来提取特征，后来使用深度学习网络进行分类。此外，我们在不同预先训练的架构上分类了我们的模型'准确性，以及像VGG16，Inceptionv3，Reset50，七峰和MobileNet V2的不同预先训练的架构中的混淆矩阵，其中VGG16和MobileNet V2更好。

从未标记数据的代表学习一直是对人工智能研究的重大兴趣。虽然自我监督的言语代表学习在语音研究界受欢迎，但很少有效地对非语音音频任务进行了全面分析了音频表示学习。在本文中，我们提出了一种自我监督的音频表示学习方法，并将其应用于各种下游非语音音频任务。我们将众所周知的Wav2Vec 2.0框架结合起来，这在用于语音任务的自我监督学习中取得了成功，具有参数效率的构装体系结构。我们的自我监督的预培训可以减少三分之二的标记数据的需求。在Audioset基准测试中，我们达到平均平均精度（地图）得分为0.415，这是通过仅限音频自我监督的学习在此数据集上的新型最先进的。我们的微调符合子也超越了在几个下游任务上以监督方式预先培训的先前系统的性能。我们进一步讨论了预先培训和微调的重要设计考虑因素。

The objective of this paper is speaker recognition under noisy and unconstrained conditions.We make two key contributions. First, we introduce a very large-scale audio-visual speaker recognition dataset collected from open-source media. Using a fully automated pipeline, we curate VoxCeleb2 which contains over a million utterances from over 6,000 speakers. This is several times larger than any publicly available speaker recognition dataset.Second, we develop and compare Convolutional Neural Network (CNN) models and training strategies that can effectively recognise identities from voice under various conditions. The models trained on the VoxCeleb2 dataset surpass the performance of previous works on a benchmark dataset by a significant margin.