注释音乐节拍在繁琐的过程中是很长的。为了打击这个问题,我们为节拍跟踪和下拍估算提出了一种新的自我监督的学习借口任务。这项任务利用SPLEETER,一个音频源分离模型,将歌曲的鼓从其其余的信号分开。第一组信号用作阳性,并通过延长否定,用于对比学习预培训。另一方面,鼓的信号用作锚点。使用此借口任务进行全卷积和复发模型时,学习了一个开始功能。在某些情况下,发现此功能被映射到歌曲中的周期元素。我们发现,当一个节拍跟踪训练集非常小(少于10个示例)时,预先训练的模型随机初始化模型表现优于随机初始化的模型。当不是这种情况时,预先训练导致了一个学习速度,导致模型过度训练集。更一般地说,这项工作定义了音乐自我监督学习领域的新观点。尤其是使用音频源分离作为自我监督的基本分量的作品之一。
translated by 谷歌翻译
The International Workshop on Reading Music Systems (WoRMS) is a workshop that tries to connect researchers who develop systems for reading music, such as in the field of Optical Music Recognition, with other researchers and practitioners that could benefit from such systems, like librarians or musicologists. The relevant topics of interest for the workshop include, but are not limited to: Music reading systems; Optical music recognition; Datasets and performance evaluation; Image processing on music scores; Writer identification; Authoring, editing, storing and presentation systems for music scores; Multi-modal systems; Novel input-methods for music to produce written music; Web-based Music Information Retrieval services; Applications and projects; Use-cases related to written music. These are the proceedings of the 3rd International Workshop on Reading Music Systems, held in Alicante on the 23rd of July 2021.
translated by 谷歌翻译
Music discovery services let users identify songs from short mobile recordings. These solutions are often based on Audio Fingerprinting, and rely more specifically on the extraction of spectral peaks in order to be robust to a number of distortions. Few works have been done to study the robustness of these algorithms to background noise captured in real environments. In particular, AFP systems still struggle when the signal to noise ratio is low, i.e when the background noise is strong. In this project, we tackle this problematic with Deep Learning. We test a new hybrid strategy which consists of inserting a denoising DL model in front of a peak-based AFP algorithm. We simulate noisy music recordings using a realistic data augmentation pipeline, and train a DL model to denoise them. The denoising model limits the impact of background noise on the AFP system's extracted peaks, improving its robustness to noise. We further propose a novel loss function to adapt the DL model to the considered AFP system, increasing its precision in terms of retrieved spectral peaks. To the best of our knowledge, this hybrid strategy has not been tested before.
translated by 谷歌翻译
在实践中,非常苛刻,有时无法收集足够大的标记数据数据集以成功培训机器学习模型,并且对此问题的一个可能解决方案是转移学习。本研究旨在评估如何可转让的时间序列数据和哪些条件下的不同域之间的特征。在训练期间,在模型的预测性能和收敛速度方面观察到转移学习的影响。在我们的实验中,我们使用1,500和9,000个数据实例的减少数据集来模仿现实世界的条件。使用相同的缩小数据集,我们培训了两组机器学习模型:那些随着转移学习的培训和从头开始培训的机器学习模型。使用四台机器学习模型进行实验。在相同的应用领域(地震学)以及相互不同的应用领域(地震,语音,医学,金融)之间进行知识转移。我们在训练期间遵守模型的预测性能和收敛速度。为了确认所获得的结果的有效性,我们重复了实验七次并应用了统计测试以确认结果的重要性。我们研究的一般性结论是转移学习可能会增加或不会对模型的预测性能或其收敛速度产生负面影响。在更多细节中分析收集的数据,以确定哪些源域和目标域兼容以用于传输知识。我们还分析了目标数据集大小的效果和模型的选择及其超参数对转移学习的影响。
translated by 谷歌翻译
最近,自我监督的表示学习(SSRL)在计算机视觉,语音,自然语言处理(NLP)以及最近的其他类型的模式(包括传感器的时间序列)中引起了很多关注。自我监督学习的普及是由传统模型通常需要大量通知数据进行培训的事实所驱动的。获取带注释的数据可能是一个困难且昂贵的过程。已经引入了自我监督的方法,以通过使用从原始数据自由获得的监督信号对模型进行判别预训练来提高训练数据的效率。与现有的对SSRL的评论不同,该评论旨在以单一模式为重点介绍CV或NLP领域的方法,我们旨在为时间数据提供对多模式自我监督学习方法的首次全面审查。为此,我们1)提供现有SSRL方法的全面分类,2)通过定义SSRL框架的关键组件来引入通用管道,3)根据其目标功能,网络架构和潜在应用程序,潜在的应用程序,潜在的应用程序,比较现有模型, 4)查看每个类别和各种方式中的现有多模式技术。最后,我们提出了现有的弱点和未来的机会。我们认为,我们的工作对使用多模式和/或时间数据的域中SSRL的要求有了一个观点
translated by 谷歌翻译
Through solving pretext tasks, self-supervised learning leverages unlabeled data to extract useful latent representations replacing traditional input features in the downstream task. In audio/speech signal processing, a wide range of features where engineered through decades of research efforts. As it turns out, learning to predict such features (a.k.a pseudo-labels) has proven to be a particularly relevant pretext task, leading to useful self-supervised representations which prove to be effective for downstream tasks. However, methods and common practices for combining such pretext tasks for better performance on the downstream task have not been explored and understood properly. In fact, the process relies almost exclusively on a computationally heavy experimental procedure, which becomes intractable with the increase of the number of pretext tasks. This paper introduces a method to select a group of pretext tasks among a set of candidates. The method we propose estimates calibrated weights for the partial losses corresponding to the considered pretext tasks during the self-supervised training process. The experiments conducted on automatic speech recognition, speaker and emotion recognition validate our approach, as the groups selected and weighted with our method perform better than classic baselines, thus facilitating the selection and combination of relevant pseudo-labels for self-supervised representation learning.
translated by 谷歌翻译
受到计算机视觉的自我监督学习的最新进展的启发,在本文中,我们介绍了Delores,这是一种新的通用音频表示方法。我们的主要目标是使我们的网络学习在资源受限的设置(数据和计算)中,可以很好地跨越各种下游任务。受Barlow Twins目标功能的启发,我们建议学习对输入音频样本失真不变的嵌入,同时确保它们包含有关样本的非冗余信息。为此,我们测量了两个相同的网络的输出之间的互相关矩阵,该网络用从音频文件采样的音频段的变形版本中,使其尽可能接近身份矩阵。我们将大规模音频集数据集和FSD50K的一小部分组合用于自学学习,并且与最先进的算法相比,参数的一半不到一半。为了进行评估,我们将这些学习的表示形式转移到9个下游分类任务,包括语音,音乐和动物声音,并在不同的评估设置下显示竞争结果。除了简单明了,我们的预训练算法还可以通过其固有的构造本质来计算,并且不需要仔细的实施细节以避免琐碎或退化的解决方案。此外,我们对结果进行消融研究,并使我们的所有代码和预培训模型公开可用https://github.com/speech-lab-iitm/delores。
translated by 谷歌翻译
在深度学习研究中,自学学习(SSL)引起了极大的关注,引起了计算机视觉和遥感社区的兴趣。尽管计算机视觉取得了很大的成功,但SSL在地球观测领域的大部分潜力仍然锁定。在本文中,我们对在遥感的背景下为计算机视觉的SSL概念和最新发展提供了介绍,并回顾了SSL中的概念和最新发展。此外,我们在流行的遥感数据集上提供了现代SSL算法的初步基准,从而验证了SSL在遥感中的潜力,并提供了有关数据增强的扩展研究。最后,我们确定了SSL未来研究的有希望的方向的地球观察(SSL4EO),以铺平了两个领域的富有成效的相互作用。
translated by 谷歌翻译
自从几十年前的频谱分析开创性工作以来,已经研究了提取音频和语音特征的方法。最近的努力以开发通用音频表示的雄心为指导。例如,如果深度神经网络在大型音频数据集上进行了培训,则可以提取最佳的嵌入。这项工作扩展了基于自我监督的学习,通过引导,提出各种编码器体系结构,并探索使用不同的预训练数据集的效果。最后,我们提出了一个新颖的培训框架,以提出一个混合音频表示,该框架结合了手工制作和数据驱动的学习音频功能。在HEAR NEURIPS 2021挑战中,对听觉场景分类和时间戳检测任务进行了评估。我们的结果表明,在大多数听到挑战任务中,带有卷积变压器的混合模型都会产生卓越的性能。
translated by 谷歌翻译
海洋生态系统及其鱼类栖息地越来越重要,因为它们在提供有价值的食物来源和保护效果方面的重要作用。由于它们的偏僻且难以接近自然,因此通常使用水下摄像头对海洋环境和鱼类栖息地进行监测。这些相机产生了大量数字数据,这些数据无法通过当前的手动处理方法有效地分析,这些方法涉及人类观察者。 DL是一种尖端的AI技术,在分析视觉数据时表现出了前所未有的性能。尽管它应用于无数领域,但仍在探索其在水下鱼类栖息地监测中的使用。在本文中,我们提供了一个涵盖DL的关键概念的教程,该教程可帮助读者了解对DL的工作原理的高级理解。该教程还解释了一个逐步的程序,讲述了如何为诸如水下鱼类监测等挑战性应用开发DL算法。此外,我们还提供了针对鱼类栖息地监测的关键深度学习技术的全面调查,包括分类,计数,定位和细分。此外,我们对水下鱼类数据集进行了公开调查,并比较水下鱼类监测域中的各种DL技术。我们还讨论了鱼类栖息地加工深度学习的新兴领域的一些挑战和机遇。本文是为了作为希望掌握对DL的高级了解,通过遵循我们的分步教程而为其应用开发的海洋科学家的教程,并了解如何发展其研究,以促进他们的研究。努力。同时,它适用于希望调查基于DL的最先进方法的计算机科学家,以进行鱼类栖息地监测。
translated by 谷歌翻译
Videos are a rich source of multi-modal supervision. In this work, we learn representations using self-supervision by leveraging three modalities naturally present in videos: visual, audio and language streams. To this end, we introduce the notion of a multimodal versatile network -a network that can ingest multiple modalities and whose representations enable downstream tasks in multiple modalities. In particular, we explore how best to combine the modalities, such that fine-grained representations of the visual and audio modalities can be maintained, whilst also integrating text into a common embedding. Driven by versatility, we also introduce a novel process of deflation, so that the networks can be effortlessly applied to the visual data in the form of video or a static image. We demonstrate how such networks trained on large collections of unlabelled video data can be applied on video, video-text, image and audio tasks. Equipped with these representations, we obtain state-of-the-art performance on multiple challenging benchmarks including UCF101, HMDB51, Kinetics600, Audioset and ESC-50 when compared to previous self-supervised work. Our models are publicly available [1, 2, 3]. * Equal contribution. † Work done during an internship at DeepMind. 34th Conference on Neural Information Processing Systems (NeurIPS 2020),
translated by 谷歌翻译
自我监督学习(SSL)是一个新的范式,用于学习判别性表示没有标记的数据,并且与受监督的对手相比,已经达到了可比甚至最新的结果。对比度学习(CL)是SSL中最著名的方法之一,试图学习一般性的信息表示数据。 CL方法主要是针对仅使用单个传感器模态的计算机视觉和自然语言处理应用程序开发的。但是,大多数普遍的计算应用程序都从各种不同的传感器模式中利用数据。虽然现有的CL方法仅限于从一个或两个数据源学习,但我们提出了可可(Crockoa)(交叉模态对比度学习),这是一种自我监督的模型,该模型采用新颖的目标函数来通过计算多功能器数据来学习质量表示形式不同的数据方式,并最大程度地减少了无关实例之间的相似性。我们评估可可对八个最近引入最先进的自我监督模型的有效性,以及五个公共数据集中的两个受监督的基线。我们表明,可可与所有其他方法相比,可可的分类表现出色。同样,可可比其他可用标记数据的十分之一的基线(包括完全监督的模型)的标签高得多。
translated by 谷歌翻译
We consider the question: what can be learnt by looking at and listening to a large number of unlabelled videos? There is a valuable, but so far untapped, source of information contained in the video itself -the correspondence between the visual and the audio streams, and we introduce a novel "Audio-Visual Correspondence" learning task that makes use of this. Training visual and audio networks from scratch, without any additional supervision other than the raw unconstrained videos themselves, is shown to successfully solve this task, and, more interestingly, result in good visual and audio representations. These features set the new state-of-the-art on two sound classification benchmarks, and perform on par with the state-of-the-art selfsupervised approaches on ImageNet classification. We also demonstrate that the network is able to localize objects in both modalities, as well as perform fine-grained recognition tasks.
translated by 谷歌翻译
In recent years, deep learning has infiltrated every field it has touched, reducing the need for specialist knowledge and automating the process of knowledge discovery from data. This review argues that astronomy is no different, and that we are currently in the midst of a deep learning revolution that is transforming the way we do astronomy. We trace the history of astronomical connectionism from the early days of multilayer perceptrons, through the second wave of convolutional and recurrent neural networks, to the current third wave of self-supervised and unsupervised deep learning. We then predict that we will soon enter a fourth wave of astronomical connectionism, in which finetuned versions of an all-encompassing 'foundation' model will replace expertly crafted deep learning models. We argue that such a model can only be brought about through a symbiotic relationship between astronomy and connectionism, whereby astronomy provides high quality multimodal data to train the foundation model, and in turn the foundation model is used to advance astronomical research.
translated by 谷歌翻译
State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories. This restricted form of supervision limits their generality and usability since additional labeled data is needed to specify any other visual concept. Learning directly from raw text about images is a promising alternative which leverages a much broader source of supervision. We demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400 million (image, text) pairs collected from the internet. After pre-training, natural language is used to reference learned visual concepts (or describe new ones) enabling zero-shot transfer of the model to downstream tasks. We study the performance of this approach by benchmarking on over 30 different existing computer vision datasets, spanning tasks such as OCR, action recognition in videos, geo-localization, and many types of fine-grained object classification. The model transfers non-trivially to most tasks and is often competitive with a fully supervised baseline without the need for any dataset specific training. For instance, we match the accuracy of the original ResNet-50 on ImageNet zero-shot without needing to use any of the 1.28 million training examples it was trained on. We release our code and pre-trained model weights at https://github.com/OpenAI/CLIP.
translated by 谷歌翻译
尽管深度强化学习(RL)最近取得了许多成功,但其方法仍然效率低下,这使得在数据方面解决了昂贵的许多问题。我们的目标是通过利用未标记的数据中的丰富监督信号来进行学习状态表示,以解决这一问题。本文介绍了三种不同的表示算法,可以访问传统RL算法使用的数据源的不同子集使用:(i)GRICA受到独立组件分析(ICA)的启发,并训练深层神经网络以输出统计独立的独立特征。输入。 Grica通过最大程度地减少每个功能与其他功能之间的相互信息来做到这一点。此外,格里卡仅需要未分类的环境状态。 (ii)潜在表示预测(LARP)还需要更多的上下文:除了要求状态作为输入外,它还需要先前的状态和连接它们的动作。该方法通过预测当前状态和行动的环境的下一个状态来学习状态表示。预测器与图形搜索算法一起使用。 (iii)重新培训通过训练深层神经网络来学习国家表示,以学习奖励功能的平滑版本。该表示形式用于预处理输入到深度RL,而奖励预测指标用于奖励成型。此方法仅需要环境中的状态奖励对学习表示表示。我们发现,每种方法都有其优势和缺点,并从我们的实验中得出结论,包括无监督的代表性学习在RL解决问题的管道中可以加快学习的速度。
translated by 谷歌翻译
在过去几年中,无监督的学习取得了很大的进展,特别是通过对比的自我监督学习。用于基准测试自我监督学习的主导数据集已经想象,最近的方法正在接近通过完全监督培训实现的性能。然而,ImageNet DataSet在很大程度上是以对象为中心的,并且目前尚不清楚这些方法的广泛不同的数据集和任务,这些方法是非以对象为中心的,例如数字病理学。虽然自我监督的学习已经开始在这个领域探讨了令人鼓舞的结果,但有理由看起来更接近这个环境与自然图像和想象成的不同。在本文中,我们对组织病理学进行了对比学学习的深入分析,引脚指向对比物镜的表现如何不同,由于组织病理学数据的特征。我们提出了一些考虑因素,例如对比目标和超参数调整的观点。在大量的实验中,我们分析了组织分类的下游性能如何受到这些考虑因素的影响。结果指出了对比学习如何减少数字病理中的注释工作,但需要考虑特定的数据集特征。为了充分利用对比学习目标,需要不同的视野和超参数校准。我们的结果为实现组织病理学应用的自我监督学习的全部潜力铺平了道路。
translated by 谷歌翻译
Sunquakes are seismic emissions visible on the solar surface, associated with some solar flares. Although discovered in 1998, they have only recently become a more commonly detected phenomenon. Despite the availability of several manual detection guidelines, to our knowledge, the astrophysical data produced for sunquakes is new to the field of Machine Learning. Detecting sunquakes is a daunting task for human operators and this work aims to ease and, if possible, to improve their detection. Thus, we introduce a dataset constructed from acoustic egression-power maps of solar active regions obtained for Solar Cycles 23 and 24 using the holography method. We then present a pedagogical approach to the application of machine learning representation methods for sunquake detection using AutoEncoders, Contrastive Learning, Object Detection and recurrent techniques, which we enhance by introducing several custom domain-specific data augmentation transformations. We address the main challenges of the automated sunquake detection task, namely the very high noise patterns in and outside the active region shadow and the extreme class imbalance given by the limited number of frames that present sunquake signatures. With our trained models, we find temporal and spatial locations of peculiar acoustic emission and qualitatively associate them to eruptive and high energy emission. While noting that these models are still in a prototype stage and there is much room for improvement in metrics and bias levels, we hypothesize that their agreement on example use cases has the potential to enable detection of weak solar acoustic manifestations.
translated by 谷歌翻译
Jitendra Malik once said, "Supervision is the opium of the AI researcher". Most deep learning techniques heavily rely on extreme amounts of human labels to work effectively. In today's world, the rate of data creation greatly surpasses the rate of data annotation. Full reliance on human annotations is just a temporary means to solve current closed problems in AI. In reality, only a tiny fraction of data is annotated. Annotation Efficient Learning (AEL) is a study of algorithms to train models effectively with fewer annotations. To thrive in AEL environments, we need deep learning techniques that rely less on manual annotations (e.g., image, bounding-box, and per-pixel labels), but learn useful information from unlabeled data. In this thesis, we explore five different techniques for handling AEL.
translated by 谷歌翻译
The remarkable success of deep learning in various domains relies on the availability of large-scale annotated datasets. However, obtaining annotations is expensive and requires great effort, which is especially challenging for videos. Moreover, the use of human-generated annotations leads to models with biased learning and poor domain generalization and robustness. As an alternative, self-supervised learning provides a way for representation learning which does not require annotations and has shown promise in both image and video domains. Different from the image domain, learning video representations are more challenging due to the temporal dimension, bringing in motion and other environmental dynamics. This also provides opportunities for video-exclusive ideas that advance self-supervised learning in the video and multimodal domain. In this survey, we provide a review of existing approaches on self-supervised learning focusing on the video domain. We summarize these methods into four different categories based on their learning objectives: 1) pretext tasks, 2) generative learning, 3) contrastive learning, and 4) cross-modal agreement. We further introduce the commonly used datasets, downstream evaluation tasks, insights into the limitations of existing works, and the potential future directions in this area.
translated by 谷歌翻译