从语音音频中删除背景噪音一直是大量研究和努力的主题，尤其是由于虚拟沟通和业余声音录制的兴起，近年来。然而，背景噪声并不是唯一可以防止可理解性的不愉快干扰：混响，剪裁，编解码器工件，有问题的均衡，有限的带宽或不一致的响度同样令人不安且无处不在。在这项工作中，我们建议将言语增强的任务视为一项整体努力，并提出了一种普遍的语音增强系统，同时解决了55种不同的扭曲。我们的方法由一种使用基于得分的扩散的生成模型以及一个多分辨率调节网络，该网络通过混合密度网络进行增强。我们表明，这种方法在专家听众执行的主观测试中大大优于艺术状态。我们还表明，尽管没有考虑任何特定的快速采样策略，但它仅通过4-8个扩散步骤就可以实现竞争性的目标得分。我们希望我们的方法论和技术贡献都鼓励研究人员和实践者采用普遍的语音增强方法，可能将其作为一项生成任务。

最近，基于扩散的生成模型已引入语音增强的任务。干净的语音损坏被建模为固定的远期过程，其中逐渐添加了越来越多的噪声。通过学习以嘈杂的输入为条件的迭代方式扭转这一过程，可以产生干净的语音。我们以先前的工作为基础，并在随机微分方程的形式主义中得出训练任务。我们对基础分数匹配目标进行了详细的理论综述，并探索了不同的采样器配置，以解决测试时的反向过程。通过使用自然图像生成文献的复杂网络体系结构，与以前的出版物相比，我们可以显着提高性能。我们还表明，我们可以与最近的判别模型竞争，并在评估与培训不同的语料库时获得更好的概括。我们通过主观的听力测试对评估结果进行补充，其中我们提出的方法是最好的。此外，我们表明所提出的方法在单渠道语音覆盖中实现了出色的最新性能。我们的代码和音频示例可在线获得，请参见https://uhh.de/inf-sp-sgmse

最近在各种语音域应用中提出了卷积增强的变压器（构象异构体），例如自动语音识别（ASR）和语音分离，因为它们可以捕获本地和全球依赖性。在本文中，我们提出了一个基于构型的度量生成对抗网络（CMGAN），以在时间频率（TF）域中进行语音增强（SE）。发电机使用两阶段构象体块编码大小和复杂的频谱图信息，以模拟时间和频率依赖性。然后，解码器将估计分解为尺寸掩模的解码器分支，以滤除不需要的扭曲和复杂的细化分支，以进一步改善幅度估计并隐式增强相信息。此外，我们还包括一个度量歧视器来通过优化相应的评估评分来减轻度量不匹配。客观和主观评估表明，与三个语音增强任务（DeNoising，dereverberation和Super-Losity）中的最新方法相比，CMGAN能够表现出卓越的性能。例如，对语音库+需求数据集的定量降解分析表明，CMGAN的表现优于以前的差距，即PESQ为3.41，SSNR为11.10 dB。

生成的对抗网络最近在神经声音中表现出了出色的表现，表现优于最佳自动回归和基于流动的模型。在本文中，我们表明这种成功可以扩展到有条件音频的其他任务。特别是，在HIFI Vocoders的基础上，我们为带宽扩展和语音增强的新型HIFI ++一般框架提出了新颖的一般框架。我们表明，通过改进的生成器体系结构和简化的多歧视培训，HIFI ++在这些任务中的最先进的情况下表现更好或与之相提并论，同时花费大量的计算资源。通过一系列广泛的实验，我们的方法的有效性得到了验证。

我们提出了一个录音录音录音的录音录音。我们的模型通过短时傅立叶变换（STFT）将其输入转换为时频表示，并使用卷积神经网络处理所得的复杂频谱图。该网络在合成音乐数据集上培训了重建和对抗性目标，该数据集是通过将干净的音乐与从旧唱片的安静片段中提取的真实噪声样本混合而创建的。我们在合成数据集的持有测试示例中定量评估我们的方法，并通过人类对实际历史记录样本的评级进行定性评估。我们的结果表明，所提出的方法可有效消除噪音，同时保留原始音乐的质量和细节。

Diffusion-based generative models have had a high impact on the computer vision and speech processing communities these past years. Besides data generation tasks, they have also been employed for data restoration tasks like speech enhancement and dereverberation. While discriminative models have traditionally been argued to be more powerful e.g. for speech enhancement, generative diffusion approaches have recently been shown to narrow this performance gap considerably. In this paper, we systematically compare the performance of generative diffusion models and discriminative approaches on different speech restoration tasks. For this, we extend our prior contributions on diffusion-based speech enhancement in the complex time-frequency domain to the task of bandwith extension. We then compare it to a discriminatively trained neural network with the same network architecture on three restoration tasks, namely speech denoising, dereverberation and bandwidth extension. We observe that the generative approach performs globally better than its discriminative counterpart on all tasks, with the strongest benefit for non-additive distortion models, like in dereverberation and bandwidth extension. Code and audio examples can be found online at https://uhh.de/inf-sp-sgmsemultitask

Diffusion models have shown a great ability at bridging the performance gap between predictive and generative approaches for speech enhancement. We have shown that they may even outperform their predictive counterparts for non-additive corruption types or when they are evaluated on mismatched conditions. However, diffusion models suffer from a high computational burden, mainly as they require to run a neural network for each reverse diffusion step, whereas predictive approaches only require one pass. As diffusion models are generative approaches they may also produce vocalizing and breathing artifacts in adverse conditions. In comparison, in such difficult scenarios, predictive models typically do not produce such artifacts but tend to distort the target speech instead, thereby degrading the speech quality. In this work, we present a stochastic regeneration approach where an estimate given by a predictive model is provided as a guide for further diffusion. We show that the proposed approach uses the predictive model to remove the vocalizing and breathing artifacts while producing very high quality samples thanks to the diffusion model, even in adverse conditions. We further show that this approach enables to use lighter sampling schemes with fewer diffusion steps without sacrificing quality, thus lifting the computational burden by an order of magnitude. Source code and audio examples are available online (https://uhh.de/inf-sp-storm).

基于分数的生成模型（SGM）最近显示了难以生成的任务的令人印象深刻的结果，例如自然图像和音频信号的无条件生成和条件生成。在这项工作中，我们将这些模型扩展到复杂的短时傅立叶变换（STFT）域，并提出了使用复杂值的深神经网络来增强语音的新型训练任务。我们在随机微分方程（SDE）的形式主义中得出了这项训练任务，从而实现了预测器 - 矫正器采样器的使用。我们提供了以前出版物启发的替代配方，以使用生成扩散模型来增强语音，从而避免了对噪声分布的任何先前假设的需求，并使训练任务纯粹是生成纯生成的，这是我们所显示的，从而改善了增强性能。

Denoising Diffusion Probabilistic Models (DDPMs) are emerging in text-to-speech (TTS) synthesis because of their strong capability of generating high-fidelity samples. However, their iterative refinement process in high-dimensional data space results in slow inference speed, which restricts their application in real-time systems. Previous works have explored speeding up by minimizing the number of inference steps but at the cost of sample quality. In this work, to improve the inference speed for DDPM-based TTS model while achieving high sample quality, we propose ResGrad, a lightweight diffusion model which learns to refine the output spectrogram of an existing TTS model (e.g., FastSpeech 2) by predicting the residual between the model output and the corresponding ground-truth speech. ResGrad has several advantages: 1) Compare with other acceleration methods for DDPM which need to synthesize speech from scratch, ResGrad reduces the complexity of task by changing the generation target from ground-truth mel-spectrogram to the residual, resulting into a more lightweight model and thus a smaller real-time factor. 2) ResGrad is employed in the inference process of the existing TTS model in a plug-and-play way, without re-training this model. We verify ResGrad on the single-speaker dataset LJSpeech and two more challenging datasets with multiple speakers (LibriTTS) and high sampling rate (VCTK). Experimental results show that in comparison with other speed-up methods of DDPMs: 1) ResGrad achieves better sample quality with the same inference speed measured by real-time factor; 2) with similar speech quality, ResGrad synthesizes speech faster than baseline methods by more than 10 times. Audio samples are available at https://resgrad1.github.io/.

In this work, we propose DiffWave, a versatile diffusion probabilistic model for conditional and unconditional waveform generation. The model is non-autoregressive, and converts the white noise signal into structured waveform through a Markov chain with a constant number of steps at synthesis. It is efficiently trained by optimizing a variant of variational bound on the data likelihood. DiffWave produces high-fidelity audio in different waveform generation tasks, including neural vocoding conditioned on mel spectrogram, class-conditional generation, and unconditional generation. We demonstrate that DiffWave matches a strong WaveNet vocoder in terms of speech quality (MOS: 4.44 versus 4.43), while synthesizing orders of magnitude faster. In particular, it significantly outperforms autoregressive and GAN-based waveform models in the challenging unconditional generation task in terms of audio quality and sample diversity from various automatic and human evaluations. 1 * Contributed to the work during an internship at Baidu Research, USA. 1 Audio samples are in: https://diffwave-demo.github.io/

DeNoising扩散模型代表了计算机视觉中最新的主题，在生成建模领域表现出了显着的结果。扩散模型是一个基于两个阶段的深层生成模型，一个正向扩散阶段和反向扩散阶段。在正向扩散阶段，通过添加高斯噪声，输入数据在几个步骤中逐渐受到干扰。在反向阶段，模型的任务是通过学习逐步逆转扩散过程来恢复原始输入数据。尽管已知的计算负担，即由于采样过程中涉及的步骤数量，扩散模型对生成样品的质量和多样性得到了广泛赞赏。在这项调查中，我们对视觉中应用的denoising扩散模型的文章进行了全面综述，包括该领域的理论和实际贡献。首先，我们识别并介绍了三个通用扩散建模框架，这些框架基于扩散概率模型，噪声调节得分网络和随机微分方程。我们进一步讨论了扩散模型与其他深层生成模型之间的关系，包括变异自动编码器，生成对抗网络，基于能量的模型，自回归模型和正常流量。然后，我们介绍了计算机视觉中应用的扩散模型的多角度分类。最后，我们说明了扩散模型的当前局限性，并设想了一些有趣的未来研究方向。

Deep neural networks (DNN) techniques have become pervasive in domains such as natural language processing and computer vision. They have achieved great success in these domains in task such as machine translation and image generation. Due to their success, these data driven techniques have been applied in audio domain. More specifically, DNN models have been applied in speech enhancement domain to achieve denosing, dereverberation and multi-speaker separation in monaural speech enhancement. In this paper, we review some dominant DNN techniques being employed to achieve speech separation. The review looks at the whole pipeline of speech enhancement from feature extraction, how DNN based tools are modelling both global and local features of speech and model training (supervised and unsupervised). We also review the use of speech-enhancement pre-trained models to boost speech enhancement process. The review is geared towards covering the dominant trends with regards to DNN application in speech enhancement in speech obtained via a single speaker.

尽管在基于生成的对抗网络（GAN）的声音编码器中，该模型在MEL频谱图中生成原始波形，但在各种录音环境中为众多扬声器合成高保真音频仍然具有挑战性。在这项工作中，我们介绍了Bigvgan，这是一款通用的Vocoder，在零照片环境中在各种看不见的条件下都很好地概括了。我们将周期性的非线性和抗氧化表现引入到发电机中，这带来了波形合成所需的感应偏置，并显着提高了音频质量。根据我们改进的生成器和最先进的歧视器，我们以最大的规模训练我们的Gan Vocoder，最高到1.12亿个参数，这在文献中是前所未有的。特别是，我们识别并解决了该规模特定的训练不稳定性，同时保持高保真输出而不过度验证。我们的Bigvgan在各种分布场景中实现了最先进的零拍性能，包括新的扬声器，新颖语言，唱歌声音，音乐和乐器音频，在看不见的（甚至是嘈杂）的录制环境中。我们将在以下网址发布我们的代码和模型：https：//github.com/nvidia/bigvgan

Music discovery services let users identify songs from short mobile recordings. These solutions are often based on Audio Fingerprinting, and rely more specifically on the extraction of spectral peaks in order to be robust to a number of distortions. Few works have been done to study the robustness of these algorithms to background noise captured in real environments. In particular, AFP systems still struggle when the signal to noise ratio is low, i.e when the background noise is strong. In this project, we tackle this problematic with Deep Learning. We test a new hybrid strategy which consists of inserting a denoising DL model in front of a peak-based AFP algorithm. We simulate noisy music recordings using a realistic data augmentation pipeline, and train a DL model to denoise them. The denoising model limits the impact of background noise on the AFP system's extracted peaks, improving its robustness to noise. We further propose a novel loss function to adapt the DL model to the considered AFP system, increasing its precision in terms of retrieved spectral peaks. To the best of our knowledge, this hybrid strategy has not been tested before.

语音转换是一项常见的语音综合任务，可以根据特定的现实情况来以不同的方式解决。最具挑战性的人通常被称为单一镜头多次的语音转换是在最一般的情况下，从一个参考语音中复制目标语音，而源和目标扬声器都不属于培训数据集。我们提出了一种基于扩散概率建模的可扩展高质量解决方案，与最新的单发语音转换方法相比，它表现出了优质的质量。此外，我们专注于实时应用程序，我们研究了可以更快地使扩散模型的一般原则，同时将合成质量保持在高水平。结果，我们开发了一种新型的随机微分方程求解器，适用于各种扩散模型类型和生成任务，如经验研究所示，并通过理论分析证明了它。

过去十年已经开发了各种各样的深度生成模型。然而，这些模型通常同时努力解决三个关键要求，包括：高样本质量，模式覆盖和快速采样。我们称之为这些要求所征收的挑战是生成的学习Trielemma，因为现有模型经常为他人交易其中一些。特别是，去噪扩散模型表明了令人印象深刻的样本质量和多样性，但它们昂贵的采样尚未允许它们在许多现实世界应用中应用。在本文中，我们认为这些模型中的缓慢采样基本上归因于去噪步骤中的高斯假设，这些假设仅针对小型尺寸的尺寸。为了使得具有大步骤的去噪，从而减少去噪步骤的总数，我们建议使用复杂的多模态分布来模拟去噪分布。我们引入了去噪扩散生成的对抗网络（去噪扩散GANS），其使用多模式条件GaN模拟每个去噪步骤。通过广泛的评估，我们表明去噪扩散GAN获得原始扩散模型的样本质量和多样性，而在CIFAR-10数据集中是2000 $ \时代。与传统的GAN相比，我们的模型表现出更好的模式覆盖和样本多样性。据我们所知，去噪扩散GaN是第一模型，可在扩散模型中降低采样成本，以便允许它们廉价地应用于现实世界应用。项目页面和代码：https://nvlabs.github.io/denoising-diffusion-gan

事实证明，神经网络是以非常低的比特率解决语音编码问题的强大工具。但是，可以在现实世界中可以强大操作的神经编码器的设计仍然是一个重大挑战。因此，我们提出了神经末端2端语音编解码器（NESC），可用于3 kbps的高质量宽带语音编码的稳定，可扩展的端到端神经语音编解码器。编码器使用一种新的体系结构配置，该配置依赖于我们提出的双PATHCONVRNN（DPCRNN）层，而解码器体系结构基于我们以前的工作streamwise-stylemelgan。我们对干净和嘈杂的语音的主观听力测试表明，NESC对于看不见的条件和信号扰动特别强大。

基于扩散的生成模型已经证明了感知上令人印象深刻的合成能力，但是它们也可以是基于可能性的模型吗？我们以肯定的方式回答了这一点，并介绍了一个基于扩散的生成模型家族，该模型可以在标准图像密度估计基准上获得最先进的可能性。与其他基于扩散的模型不同，我们的方法允许与其他模型的其余部分共同对噪声时间表进行有效优化。我们表明，根据扩散数据的信噪比，变异下限（VLB）简化为非常短的表达，从而改善了我们对该模型类别的理论理解。使用这种见解，我们证明了文献中提出的几个模型之间的等效性。此外，我们表明连续时间VLB在噪声方面不变，除了其端点处的信噪比。这使我们能够学习一个噪声时间表，以最大程度地减少所得VLB估计器的差异，从而更快地优化。将这些进步与建筑改进相结合，我们获得了图像密度估计基准的最先进的可能性，超过了多年来主导这些基准测试的自回旋模型，通常优化了很多年。此外，我们展示了如何将模型用作BITS背包压缩方案的一部分，并展示了接近理论最佳的无损压缩率。代码可在https://github.com/google-research/vdm上找到。

Prior works on improving speech quality with visual input typically study each type of auditory distortion separately (e.g., separation, inpainting, video-to-speech) and present tailored algorithms. This paper proposes to unify these subjects and study Generalized Speech Enhancement, where the goal is not to reconstruct the exact reference clean signal, but to focus on improving certain aspects of speech. In particular, this paper concerns intelligibility, quality, and video synchronization. We cast the problem as audio-visual speech resynthesis, which is composed of two steps: pseudo audio-visual speech recognition (P-AVSR) and pseudo text-to-speech synthesis (P-TTS). P-AVSR and P-TTS are connected by discrete units derived from a self-supervised speech model. Moreover, we utilize self-supervised audio-visual speech model to initialize P-AVSR. The proposed model is coined ReVISE. ReVISE is the first high-quality model for in-the-wild video-to-speech synthesis and achieves superior performance on all LRS3 audio-visual enhancement tasks with a single model. To demonstrates its applicability in the real world, ReVISE is also evaluated on EasyCom, an audio-visual benchmark collected under challenging acoustic conditions with only 1.6 hours of training data. Similarly, ReVISE greatly suppresses noise and improves quality. Project page: https://wnhsu.github.io/ReVISE.

尽管扩散模型在图像生成中表现出了巨大的成功，但它们的噪声生成过程并未明确考虑图像的结构，例如它们固有的多尺度性质。受扩散模型的启发和粗到精细建模的可取性，我们提出了一个新模型，该模型通过迭代反转热方程式生成图像，当在图像的2D平面上运行时，PDE局部删除了细尺度信息。在我们的新方法中，正向热方程的解被解释为有向图形模型中的变异近似。我们展示了有希望的图像质量，并指出了在扩散模型中未见的新兴定性特性，例如在神经网络可解释性的图像和各个方面的整体颜色和形状分解。对自然图像的光谱分析将我们的模型定位为扩散模型的一种双重偶，并揭示了其中的隐式感应偏见。