对音频信号的长期依赖性进行建模是一个特别具有挑战性的问题，因为即使是小型尺度的产量，也要在十万个样本上产生。随着变形金刚的最近出现，神经体系结构擅长于更长的时间尺度建模依赖性，但它们受到二次限制的限制来扩展它们。我们提出了一种生成的自动回归体系结构，该体系结构可以在相当大的上下文中对音频波形进行建模，超过500,000个样本。我们的工作适应了通过CNN前端学习潜在表示，然后使用变压器编码器，经过全面训练的端到端学习来学习时间依赖性：从而允许它认为适合于该表示的表示形式。下一个样本。与以前的作品比较了不同的时间量表以显示改进，我们使用标准数据集，具有相同数量的参数/上下文来显示改进。与其他方法相比，我们在标准数据集中实现了最先进的性能，例如WaveNet，Sashmi和Sample-RNN，用于建模长期结构。这项工作为该领域提供了非常令人兴奋的方向，鉴于上下文建模的改进，可以通过使用数十亿/万亿个参数来缩放更多数据，并可能更好地结果。

本文提供了一种在没有传统艺术神经结构状态的情况下做大规模音频理解的方式。自从在过去十年中引入理解音频信号的深度学习以来，卷积架构已经能够实现最先进的艺术状态，这些结果超越了传统的手工制作功能。在最近的过去，远离传统的卷积和经常性神经网络相似的转变，朝纯端到端的变压器架构。在这项工作中，我们探索了一种基于袋式模型的方法。我们的方法没有任何卷积，复发，关注，变压器或其他方法如伯特。我们利用Micro和Macro Level群集Vanilla Embeddings，并使用MLP头进行分类。我们仅使用前馈编码器解码器模型来获取光谱包围，光谱贴片和切片以及多分辨率光谱的瓶颈。类似于SIMCLR中的方法的分类头（前馈层）在学习的表示上培训。使用简单的代码在潜在的陈述中了解到，我们展示了我们如何超越传统的卷积神经网络架构，并引人注目地接近优势强大的变压器架构。这项工作希望能够在没有大规模的端到端神经架构的情况下铺平道具的兴奋进步。

现实世界中的数据是高维的：即使在压缩后，书籍，图像或音乐表演也很容易包含数十万个元素。但是，最常用的自回归模型，变压器非常昂贵，以缩放捕获这种远程结构所需的输入和层数。我们开发了感知者AR，这是一种自回归的模态 - 不合骨架构，它使用交叉注意力将远程输入映射到少数潜在的潜在，同时还可以维护端到端的因果关系掩盖。感知器AR可以直接进行十万个令牌，从而实现了实用的长篇小写密度估计，而无需手工制作的稀疏模式或记忆机制。当对图像或音乐进行培训时，感知器AR会生成具有清晰长期连贯性和结构的输出。我们的架构还获得了长期基准测试的最新可能性，包括64 x 64个Imagenet图像和PG-19书籍。

Previous works (Donahue et al., 2018a;Engel et al., 2019a) have found that generating coherent raw audio waveforms with GANs is challenging. In this paper, we show that it is possible to train GANs reliably to generate high quality coherent waveforms by introducing a set of architectural changes and simple training techniques. Subjective evaluation metric (Mean Opinion Score, or MOS) shows the effectiveness of the proposed approach for high quality mel-spectrogram inversion. To establish the generality of the proposed techniques, we show qualitative results of our model in speech synthesis, music domain translation and unconditional music synthesis. We evaluate the various components of the model through ablation studies and suggest a set of guidelines to design general purpose discriminators and generators for conditional sequence synthesis tasks. Our model is non-autoregressive, fully convolutional, with significantly fewer parameters than competing models and generalizes to unseen speakers for mel-spectrogram inversion. Our pytorch implementation runs at more than 100x faster than realtime on GTX 1080Ti GPU and more than 2x faster than real-time on CPU, without any hardware specific optimization tricks.

This paper introduces WaveNet, a deep neural network for generating raw audio waveforms. The model is fully probabilistic and autoregressive, with the predictive distribution for each audio sample conditioned on all previous ones; nonetheless we show that it can be efficiently trained on data with tens of thousands of samples per second of audio. When applied to text-to-speech, it yields state-ofthe-art performance, with human listeners rating it as significantly more natural sounding than the best parametric and concatenative systems for both English and Mandarin. A single WaveNet can capture the characteristics of many different speakers with equal fidelity, and can switch between them by conditioning on the speaker identity. When trained to model music, we find that it generates novel and often highly realistic musical fragments. We also show that it can be employed as a discriminative model, returning promising results for phoneme recognition.

深度学习算法的兴起引领许多研究人员使用经典信号处理方法来发声。深度学习模型已经实现了富有富有的语音合成，现实的声音纹理和虚拟乐器的音符。然而，最合适的深度学习架构仍在调查中。架构的选择紧密耦合到音频表示。声音的原始波形可以太密集和丰富，用于深入学习模型，以有效处理 - 复杂性提高培训时间和计算成本。此外，它不代表声音以其所感知的方式。因此，在许多情况下，原始音频已经使用上采样，特征提取，甚至采用波形的更高级别的图示来转换为压缩和更有意义的形式。此外，研究了所选择的形式，另外的调节表示，不同的模型架构以及用于评估重建声音的许多度量的条件。本文概述了应用于使用深度学习的声音合成的音频表示。此外，它呈现了使用深度学习模型开发和评估声音合成架构的最重要方法，始终根据音频表示。

我们介绍Audiolm，这是具有长期一致性高质量音频产生的框架。 Audiolm将输入音频映射到一系列离散令牌，并将音频生成作为此表示空间中的语言建模任务。我们展示了现有的音频令牌如何在重建质量和长期结构之间提供不同的权衡，我们提出了一个混合代币化计划来实现这两个目标。也就是说，我们利用在音频中预先训练的蒙版语言模型的离散激活来捕获长期结构和神经音频编解码器产生的离散代码，以实现高质量的合成。通过培训大型原始音频波形，Audiolm学会了在简短的提示下产生自然和连贯的连续性。当接受演讲训练时，没有任何笔录或注释，Audiolm会在句法和语义上产生可行的语音连续性，同时还为看不见的说话者保持说话者身份和韵律。此外，我们演示了我们的方法如何通过产生连贯的钢琴音乐连续性来超越语音，尽管受过训练而没有任何象征性的音乐代表。

Transformers and variational autoencoders (VAE) have been extensively employed for symbolic (e.g., MIDI) domain music generation. While the former boast an impressive capability in modeling long sequences, the latter allow users to willingly exert control over different parts (e.g., bars) of the music to be generated. In this paper, we are interested in bringing the two together to construct a single model that exhibits both strengths. The task is split into two steps. First, we equip Transformer decoders with the ability to accept segment-level, time-varying conditions during sequence generation. Subsequently, we combine the developed and tested in-attention decoder with a Transformer encoder, and train the resulting MuseMorphose model with the VAE objective to achieve style transfer of long pop piano pieces, in which users can specify musical attributes including rhythmic intensity and polyphony (i.e., harmonic fullness) they desire, down to the bar level. Experiments show that MuseMorphose outperforms recurrent neural network (RNN) based baselines on numerous widely-used metrics for style transfer tasks.

生成建模研究的持续趋势是将样本分辨率推高更高，同时减少培训和采样的计算要求。我们的目标是通过技术的组合进一步推动这一趋势 - 每个组件代表当前效率在各自领域的顶峰。其中包括载体定量的GAN（VQ-GAN），该模型具有高水平的损耗 - 但感知上微不足道的压缩模型；沙漏变形金刚，一个高度可扩展的自我注意力模型；和逐步未胶片的denoising自动编码器（Sundae），一种非自动化（NAR）文本生成模型。出乎意料的是，当应用于多维数据时，我们的方法突出了沙漏变压器的原始公式中的弱点。鉴于此，我们建议对重采样机制进行修改，该机制适用于将分层变压器应用于多维数据的任何任务。此外，我们证明了圣代表到长序列长度的可伸缩性 - 比先前的工作长四倍。我们提出的框架秤达到高分辨率（$ 1024 \ times 1024 $），并迅速火车（2-4天）。至关重要的是，训练有素的模型在消费级GPU（GTX 1080TI）上大约2秒内生产多样化和现实的百像样品。通常，该框架是灵活的：支持任意数量的采样步骤，示例自动插入，自我纠正功能，有条件的生成和NAR公式，以允许任意介绍掩护。我们在FFHQ256上获得10.56的FID得分 - 仅在100个采样步骤中以不到一半的采样步骤接近原始VQ -GAN，而FFHQ1024的FFHQ1024和21.85。

Music discovery services let users identify songs from short mobile recordings. These solutions are often based on Audio Fingerprinting, and rely more specifically on the extraction of spectral peaks in order to be robust to a number of distortions. Few works have been done to study the robustness of these algorithms to background noise captured in real environments. In particular, AFP systems still struggle when the signal to noise ratio is low, i.e when the background noise is strong. In this project, we tackle this problematic with Deep Learning. We test a new hybrid strategy which consists of inserting a denoising DL model in front of a peak-based AFP algorithm. We simulate noisy music recordings using a realistic data augmentation pipeline, and train a DL model to denoise them. The denoising model limits the impact of background noise on the AFP system's extracted peaks, improving its robustness to noise. We further propose a novel loss function to adapt the DL model to the considered AFP system, increasing its precision in terms of retrieved spectral peaks. To the best of our knowledge, this hybrid strategy has not been tested before.

创造像音乐这样的复杂艺术作品需要深刻的创造力。随着深度学习和强大模型（例如变形金刚）的最新进展，自动音乐生成取得了巨大进展。在伴奏的生成环境中，在歌曲中的适当位置创建一个连贯的鼓模式，即使对于经验丰富的鼓手来说，在歌曲中的适当位置也是一项艰巨的任务。鼓节拍倾向于通过填充或即兴表演的节遵循重复的模式。在这项工作中，我们解决了鼓模式产生的任务，该任务是根据四种旋律乐器演奏的音乐来解决的：钢琴，吉他，贝斯和弦乐。我们将变压器序列用于序列模型来生成在旋律伴奏下进行的基本鼓模式，以发现即兴创作在很大程度上不存在，这可能归因于其在训练数据中的预期相对较低的表示。我们提出了一种新颖的功能，以捕获相对于其邻居的标准中即兴创作的程度。我们训练一个模型，以预测旋律伴奏曲目的即兴位置。最后，我们使用一种小说的伯特（Bert）启发的填充体系结构，以学习鼓和旋律的结构，以实现即兴音乐的填充元素。

序列建模的一个中心目标是设计一个单个原则模型，该模型可以解决各种方式和任务，尤其是在远程依赖方面的序列数据。尽管包括RNN，CNN和Transformers在内的传统模型具有用于捕获长期依赖性的专业变体，但它们仍然很难扩展到长时间的10000美元或更多步骤。通过模拟基本状态空间模型（SSM）\（x'（t）= ax（t）= ax（t） + bu（t），y（t）= cx（t） + du（t） + du（t）\ ), and showed that for appropriate choices of the state matrix \( A \), this system could handle long-range dependencies mathematically and empirically.但是，该方法具有过度的计算和内存需求，使其无法作为一般序列建模解决方案。我们根据SSM的新参数化提出了结构化状态空间序列模型（S4），并表明它可以比以前的方法更有效地计算出其理论强度。我们的技术涉及对\（a \）进行低级校正的调节，从而使其对角度稳定，并将SSM降低到库奇内核的精心研究的计算中。 S4在各种既定的基准测试范围内取得了强劲的经验结果，包括（i）在顺序CIFAR-10上的91 \％精度，没有数据增强或辅助损失，与较大的2-D Resnet相当，（ii）实质上关闭。在图像和语言建模任务上与变形金刚的差距，同时在远程竞技场基准的每个任务上执行每一代$ 60 \ times $ $（iii）sota，包括求解所有先前工作的挑战性path-x任务，而所有先前工作的长度为16K，同时与所有竞争对手一样高效。

Inspired by progress in unsupervised representation learning for natural language, we examine whether similar models can learn useful representations for images. We train a sequence Transformer to auto-regressively predict pixels, without incorporating knowledge of the 2D input structure. Despite training on low-resolution ImageNet without labels, we find that a GPT-2 scale model learns strong image representations as measured by linear probing, fine-tuning, and low-data classification. On CIFAR-10, we achieve 96.3% accuracy with a linear probe, outperforming a supervised Wide ResNet, and 99.0% accuracy with full fine-tuning, matching the top supervised pretrained models. We are also competitive with self-supervised benchmarks on ImageNet when substituting pixels for a VQVAE encoding, achieving 69.0% top-1 accuracy on a linear probe of our features.

我们听到的每种声音都是连续的卷积操作的结果（例如，室内声学，麦克风特性，仪器本身的共振特性，更不用说声音复制系统的特征和局限性了）。在这项工作中，我们试图确定使用AI执行特定作品的最佳空间。此外，我们使用房间声学作为增强给定声音的感知品质的一种方式。从历史上看，房间（尤其是教堂和音乐厅）旨在主持和提供特定的音乐功能。在某些情况下，建筑声学品质增强了那里的音乐。我们试图通过指定房间冲动响应来模仿这一步骤，这些响应与为特定音乐产生增强的声音质量相关。首先，对卷积架构进行了培训，可以采用音频样本，并模仿各种仪器家族准确性约78％的专家的评分，并具有感知品质的笔记。这为我们提供了任何音频样本的评分功能，可以自动评分音符的感知愉悦度。现在，通过一个大约有60,000个合成冲动响应的库，模仿了各种房间，材料等，我们使用简单的卷积操作来改变声音，就好像它在特定的房间里播放一样。感知评估者用于对音乐声音进行排名，并产生“最佳房间或音乐厅”来播放声音。作为副产品，它还可以使用房间声学将质量差的声音变成“好”声音。

Conventional methods for human motion synthesis are either deterministic or struggle with the trade-off between motion diversity and motion quality. In response to these limitations, we introduce MoFusion, i.e., a new denoising-diffusion-based framework for high-quality conditional human motion synthesis that can generate long, temporally plausible, and semantically accurate motions based on a range of conditioning contexts (such as music and text). We also present ways to introduce well-known kinematic losses for motion plausibility within the motion diffusion framework through our scheduled weighting strategy. The learned latent space can be used for several interactive motion editing applications -- like inbetweening, seed conditioning, and text-based editing -- thus, providing crucial abilities for virtual character animation and robotics. Through comprehensive quantitative evaluations and a perceptual user study, we demonstrate the effectiveness of MoFusion compared to the state of the art on established benchmarks in the literature. We urge the reader to watch our supplementary video and visit https://vcai.mpi-inf.mpg.de/projects/MoFusion.

Biological systems perceive the world by simultaneously processing high-dimensional inputs from modalities as diverse as vision, audition, touch, proprioception, etc. The perception models used in deep learning on the other hand are designed for individual modalities, often relying on domainspecific assumptions such as the local grid structures exploited by virtually all existing vision models. These priors introduce helpful inductive biases, but also lock models to individual modalities. In this paper we introduce the Perceiver -a model that builds upon Transformers and hence makes few architectural assumptions about the relationship between its inputs, but that also scales to hundreds of thousands of inputs, like ConvNets. The model leverages an asymmetric attention mechanism to iteratively distill inputs into a tight latent bottleneck, allowing it to scale to handle very large inputs. We show that this architecture is competitive with or outperforms strong, specialized models on classification tasks across various modalities: images, point clouds, audio, video, and video+audio. The Perceiver obtains performance comparable to ResNet-50 and ViT on ImageNet without 2D convolutions by directly attending to 50,000 pixels. It is also competitive in all modalities in AudioSet.

Diffusion models have quickly become the go-to paradigm for generative modelling of perceptual signals (such as images and sound) through iterative refinement. Their success hinges on the fact that the underlying physical phenomena are continuous. For inherently discrete and categorical data such as language, various diffusion-inspired alternatives have been proposed. However, the continuous nature of diffusion models conveys many benefits, and in this work we endeavour to preserve it. We propose CDCD, a framework for modelling categorical data with diffusion models that are continuous both in time and input space. We demonstrate its efficacy on several language modelling tasks.

Deep neural networks (DNN) techniques have become pervasive in domains such as natural language processing and computer vision. They have achieved great success in these domains in task such as machine translation and image generation. Due to their success, these data driven techniques have been applied in audio domain. More specifically, DNN models have been applied in speech enhancement domain to achieve denosing, dereverberation and multi-speaker separation in monaural speech enhancement. In this paper, we review some dominant DNN techniques being employed to achieve speech separation. The review looks at the whole pipeline of speech enhancement from feature extraction, how DNN based tools are modelling both global and local features of speech and model training (supervised and unsupervised). We also review the use of speech-enhancement pre-trained models to boost speech enhancement process. The review is geared towards covering the dominant trends with regards to DNN application in speech enhancement in speech obtained via a single speaker.

自动音频字幕是一项跨模式翻译任务，旨在为给定的音频剪辑生成自然语言描述。近年来，随着免费可用数据集的发布，该任务受到了越来越多的关注。该问题主要通过深度学习技术解决。已经提出了许多方法，例如研究不同的神经网络架构，利用辅助信息，例如关键字或句子信息来指导字幕生成，并采用了不同的培训策略，这些策略极大地促进了该领域的发展。在本文中，我们对自动音频字幕的已发表贡献进行了全面综述，从各种现有方法到评估指标和数据集。我们还讨论了公开挑战，并设想可能的未来研究方向。

音乐表达需要控制播放的笔记，以及如何执行它们。传统的音频合成器提供了详细的表达控制，但以现实主义的成本提供了详细的表达控制。黑匣子神经音频合成和连接采样器可以产生现实的音频，但有很少的控制机制。在这项工作中，我们介绍MIDI-DDSP乐器的分层模型，可以实现现实的神经音频合成和详细的用户控制。从可解释的可分辨率数字信号处理（DDSP）合成参数开始，我们推断出富有表现力性能的音符和高级属性（例如Timbre，Vibrato，Dynamics和Asticiculation）。这将创建3级层次结构（注释，性能，合成），提供个人选择在每个级别进行干预，或利用培训的前沿（表现给出备注，综合赋予绩效）进行创造性的帮助。通过定量实验和聆听测试，我们证明了该层次结构可以重建高保真音频，准确地预测音符序列的性能属性，独立地操纵给定性能的属性，以及作为完整的系统，从新颖的音符生成现实音频顺序。通过利用可解释的层次结构，具有多个粒度的粒度，MIDI-DDSP将门打开辅助工具的门，以赋予各种音乐体验的个人。