Conditional diffusion probabilistic models can model the distribution of natural images and can generate diverse and realistic samples based on given conditions. However, oftentimes their results can be unrealistic with observable color shifts and textures. We believe that this issue results from the divergence between the probabilistic distribution learned by the model and the distribution of natural images. The delicate conditions gradually enlarge the divergence during each sampling timestep. To address this issue, we introduce a new method that brings the predicted samples to the training data manifold using a pretrained unconditional diffusion model. The unconditional model acts as a regularizer and reduces the divergence introduced by the conditional model at each sampling step. We perform comprehensive experiments to demonstrate the effectiveness of our approach on super-resolution, colorization, turbulence removal, and image-deraining tasks. The improvements obtained by our method suggest that the priors can be incorporated as a general plugin for improving conditional diffusion models.

尽管许多远程成像系统旨在支持扩展视力应用，但由于大气湍流，其操作的自然障碍是退化。大气湍流通过引入模糊和几何变形而导致图像质量的显着降解。近年来，在文献中提出了各种基于深度学习的单图像缓解方法，包括基于CNN的基于CNN和基于GAN的反转方法，这些方法试图消除图像中的失真。但是，其中一些方法很难训练，并且通常无法重建面部特征并产生不切实际的结果，尤其是在高湍流的情况下。降级扩散概率模型（DDPM）最近由于其稳定的训练过程和产生高质量图像的能力而获得了一些吸引力。在本文中，我们提出了第一个基于DDPM的解决方案，用于缓解大气湍流问题。我们还提出了一种快速采样技术，用于减少条件DDPM的推理时间。对合成和现实世界数据进行了广泛的实验，以显示我们模型的重要性。为了促进进一步的研究，在审查过程之后，所有代码和验证的模型都将公开。

While deep learning-based methods for blind face restoration have achieved unprecedented success, they still suffer from two major limitations. First, most of them deteriorate when facing complex degradations out of their training data. Second, these methods require multiple constraints, e.g., fidelity, perceptual, and adversarial losses, which require laborious hyper-parameter tuning to stabilize and balance their influences. In this work, we propose a novel method named DifFace that is capable of coping with unseen and complex degradations more gracefully without complicated loss designs. The key of our method is to establish a posterior distribution from the observed low-quality (LQ) image to its high-quality (HQ) counterpart. In particular, we design a transition distribution from the LQ image to the intermediate state of a pre-trained diffusion model and then gradually transmit from this intermediate state to the HQ target by recursively applying a pre-trained diffusion model. The transition distribution only relies on a restoration backbone that is trained with $L_2$ loss on some synthetic data, which favorably avoids the cumbersome training process in existing methods. Moreover, the transition distribution can contract the error of the restoration backbone and thus makes our method more robust to unknown degradations. Comprehensive experiments show that DifFace is superior to current state-of-the-art methods, especially in cases with severe degradations. Our code and model are available at https://github.com/zsyOAOA/DifFace.

在不利天气条件下的图像恢复对各种计算机视觉应用引起了重大兴趣。最近的成功方法取决于深度神经网络架构设计（例如，具有视觉变压器）的当前进展。由最新的条件生成模型取得的最新进展的动机，我们提出了一种基于贴片的图像恢复算法，基于脱氧扩散概率模型。我们的基于贴片的扩散建模方法可以通过使用指导的DeNoising过程进行尺寸 - 不足的图像恢复，并在推理过程中对重叠贴片进行平滑的噪声估计。我们在基准数据集上经验评估了我们的模型，以进行图像，混合的降低和飞行以及去除雨滴的去除。我们展示了我们在特定天气和多天气图像恢复上实现最先进的表演的方法，并在质量上表现出对现实世界测试图像的强烈概括。

DeNoising扩散模型代表了计算机视觉中最新的主题，在生成建模领域表现出了显着的结果。扩散模型是一个基于两个阶段的深层生成模型，一个正向扩散阶段和反向扩散阶段。在正向扩散阶段，通过添加高斯噪声，输入数据在几个步骤中逐渐受到干扰。在反向阶段，模型的任务是通过学习逐步逆转扩散过程来恢复原始输入数据。尽管已知的计算负担，即由于采样过程中涉及的步骤数量，扩散模型对生成样品的质量和多样性得到了广泛赞赏。在这项调查中，我们对视觉中应用的denoising扩散模型的文章进行了全面综述，包括该领域的理论和实际贡献。首先，我们识别并介绍了三个通用扩散建模框架，这些框架基于扩散概率模型，噪声调节得分网络和随机微分方程。我们进一步讨论了扩散模型与其他深层生成模型之间的关系，包括变异自动编码器，生成对抗网络，基于能量的模型，自回归模型和正常流量。然后，我们介绍了计算机视觉中应用的扩散模型的多角度分类。最后，我们说明了扩散模型的当前局限性，并设想了一些有趣的未来研究方向。

Recent deep learning methods have achieved promising results in image shadow removal. However, their restored images still suffer from unsatisfactory boundary artifacts, due to the lack of degradation prior embedding and the deficiency in modeling capacity. Our work addresses these issues by proposing a unified diffusion framework that integrates both the image and degradation priors for highly effective shadow removal. In detail, we first propose a shadow degradation model, which inspires us to build a novel unrolling diffusion model, dubbed ShandowDiffusion. It remarkably improves the model's capacity in shadow removal via progressively refining the desired output with both degradation prior and diffusive generative prior, which by nature can serve as a new strong baseline for image restoration. Furthermore, ShadowDiffusion progressively refines the estimated shadow mask as an auxiliary task of the diffusion generator, which leads to more accurate and robust shadow-free image generation. We conduct extensive experiments on three popular public datasets, including ISTD, ISTD+, and SRD, to validate our method's effectiveness. Compared to the state-of-the-art methods, our model achieves a significant improvement in terms of PSNR, increasing from 31.69dB to 34.73dB over SRD dataset.

Generating photos satisfying multiple constraints find broad utility in the content creation industry. A key hurdle to accomplishing this task is the need for paired data consisting of all modalities (i.e., constraints) and their corresponding output. Moreover, existing methods need retraining using paired data across all modalities to introduce a new condition. This paper proposes a solution to this problem based on denoising diffusion probabilistic models (DDPMs). Our motivation for choosing diffusion models over other generative models comes from the flexible internal structure of diffusion models. Since each sampling step in the DDPM follows a Gaussian distribution, we show that there exists a closed-form solution for generating an image given various constraints. Our method can unite multiple diffusion models trained on multiple sub-tasks and conquer the combined task through our proposed sampling strategy. We also introduce a novel reliability parameter that allows using different off-the-shelf diffusion models trained across various datasets during sampling time alone to guide it to the desired outcome satisfying multiple constraints. We perform experiments on various standard multimodal tasks to demonstrate the effectiveness of our approach. More details can be found in https://nithin-gk.github.io/projectpages/Multidiff/index.html

自由格式介绍是在任意二进制掩码指定的区域中向图像中添加新内容的任务。大多数现有方法训练了一定的面具分布，这将其概括能力限制为看不见的掩模类型。此外，通过像素和知觉损失的训练通常会导致对缺失区域的简单质地扩展，而不是语义上有意义的一代。在这项工作中，我们提出重新启动：基于deno的扩散概率模型（DDPM）的内部介入方法，甚至适用于极端掩模。我们采用预定的无条件DDPM作为生成先验。为了调节生成过程，我们仅通过使用给定的图像信息对未掩盖的区域进行采样来改变反向扩散迭代。由于该技术不会修改或调节原始DDPM网络本身，因此该模型可为任何填充形式产生高质量和不同的输出图像。我们使用标准面具和极端口罩验证面部和通用图像的方法。重新粉刷优于最先进的自动回归，而GAN的方法至少在六个面具分布中进行了五个。 github存储库：git.io/repaint

扩散模型已显示出令人印象深刻的图像产生性能，并已用于各种计算机视觉任务。不幸的是，使用扩散模型的图像生成非常耗时，因为它需要数千个采样步骤。为了解决这个问题，我们在这里提出了一种新型的金字塔扩散模型，以使用训练有位置嵌入的单个分数函数从更粗的分辨率图像开始生成高分辨率图像。这使图像生成的时间效率抽样可以解决，并在资源有限的训练时也可以解决低批量的大小问题。此外，我们表明，使用单个分数函数可以有效地用于多尺度的超分辨率问题。

与生成的对抗网（GAN）相比，降级扩散概率模型（DDPM）在各种图像生成任务中取得了显着成功。关于语义图像综合的最新工作主要遵循\ emph {de exto}基于gan的方法，这可能导致生成图像的质量或多样性不令人满意。在本文中，我们提出了一个基于DDPM的新型框架，用于语义图像合成。与先前的条件扩散模型不同，将语义布局和嘈杂的图像作为输入为U-NET结构，该结构可能无法完全利用输入语义掩码中的信息，我们的框架处理语义布局和嘈杂的图像不同。它将噪声图像馈送到U-NET结构的编码器时，而语义布局通过多层空间自适应归一化操作符将语义布局馈送到解码器。为了进一步提高语义图像合成中的发电质量和语义解释性，我们介绍了无分类器的指导采样策略，该策略承认采样过程的无条件模型的得分。在三个基准数据集上进行的广泛实验证明了我们提出的方法的有效性，从而在忠诚度（FID）和多样性〜（LPIPS）方面实现了最先进的性能。

图像deBlurring是一种对给定输入图像的多种合理的解决方案是一个不适的问题。然而，大多数现有方法产生了清洁图像的确定性估计，并且训练以最小化像素级失真。已知这些指标与人类感知差，并且通常导致不切实际的重建。我们基于条件扩散模型介绍了盲脱模的替代框架。与现有技术不同，我们训练一个随机采样器，它改进了确定性预测器的输出，并且能够为给定输入产生多样化的合理重建。这导致跨多个标准基准的现有最先进方法的感知质量的显着提高。与典型的扩散模型相比，我们的预测和精致方法也能实现更有效的采样。结合仔细调整的网络架构和推理过程，我们的方法在PSNR等失真度量方面具有竞争力。这些结果表明了我们基于扩散和挑战的扩散和挑战的策略的显着优势，生产单一确定性重建的广泛使用策略。

Most existing Image Restoration (IR) models are task-specific, which can not be generalized to different degradation operators. In this work, we propose the Denoising Diffusion Null-Space Model (DDNM), a novel zero-shot framework for arbitrary linear IR problems, including but not limited to image super-resolution, colorization, inpainting, compressed sensing, and deblurring. DDNM only needs a pre-trained off-the-shelf diffusion model as the generative prior, without any extra training or network modifications. By refining only the null-space contents during the reverse diffusion process, we can yield diverse results satisfying both data consistency and realness. We further propose an enhanced and robust version, dubbed DDNM+, to support noisy restoration and improve restoration quality for hard tasks. Our experiments on several IR tasks reveal that DDNM outperforms other state-of-the-art zero-shot IR methods. We also demonstrate that DDNM+ can solve complex real-world applications, e.g., old photo restoration.

扩散模型（DMS）显示出高质量图像合成的巨大潜力。但是，当涉及到具有复杂场景的图像时，如何正确描述图像全局结构和对象细节仍然是一项具有挑战性的任务。在本文中，我们提出了弗里多（Frido），这是一种特征金字塔扩散模型，该模型执行了图像合成的多尺度粗到1个降解过程。我们的模型将输入图像分解为依赖比例的矢量量化特征，然后是用于产生图像输出的粗到细门。在上述多尺度表示阶段，可以进一步利用文本，场景图或图像布局等其他输入条件。因此，还可以将弗里多应用于条件或跨模式图像合成。我们对各种无条件和有条件的图像生成任务进行了广泛的实验，从文本到图像综合，布局到图像，场景环形图像到标签形象。更具体地说，我们在五个基准测试中获得了最先进的FID分数，即可可和开阔图像的布局到图像，可可和视觉基因组的场景环形图像以及可可的标签对图像图像。 。代码可在https://github.com/davidhalladay/frido上找到。

Diffusion Probabilistic Models (DPMs) have recently been employed for image deblurring. DPMs are trained via a stochastic denoising process that maps Gaussian noise to the high-quality image, conditioned on the concatenated blurry input. Despite their high-quality generated samples, image-conditioned Diffusion Probabilistic Models (icDPM) rely on synthetic pairwise training data (in-domain), with potentially unclear robustness towards real-world unseen images (out-of-domain). In this work, we investigate the generalization ability of icDPMs in deblurring, and propose a simple but effective guidance to significantly alleviate artifacts, and improve the out-of-distribution performance. Particularly, we propose to first extract a multiscale domain-generalizable representation from the input image that removes domain-specific information while preserving the underlying image structure. The representation is then added into the feature maps of the conditional diffusion model as an extra guidance that helps improving the generalization. To benchmark, we focus on out-of-distribution performance by applying a single-dataset trained model to three external and diverse test sets. The effectiveness of the proposed formulation is demonstrated by improvements over the standard icDPM, as well as state-of-the-art performance on perceptual quality and competitive distortion metrics compared to existing methods.

通过将图像形成过程分解成逐个申请的去噪自身额，扩散模型（DMS）实现了最先进的合成导致图像数据和超越。另外，它们的配方允许引导机构来控制图像生成过程而不会再刷新。然而，由于这些模型通常在像素空间中直接操作，因此强大的DMS的优化通常消耗数百个GPU天，并且由于顺序评估，推理是昂贵的。为了在保留其质量和灵活性的同时启用有限计算资源的DM培训，我们将它们应用于强大的佩带自动化器的潜在空间。与以前的工作相比，这种代表上的培训扩散模型允许第一次达到复杂性降低和细节保存之间的近乎最佳点，极大地提高了视觉保真度。通过将跨关注层引入模型架构中，我们将扩散模型转化为强大而柔性的发电机，以进行诸如文本或边界盒和高分辨率合成的通用调节输入，以卷积方式变得可以实现。我们的潜在扩散模型（LDMS）实现了一种新的技术状态，可在各种任务中进行图像修复和高竞争性能，包括无条件图像生成，语义场景合成和超级分辨率，同时与基于像素的DMS相比显着降低计算要求。代码可在https://github.com/compvis/lattent-diffusion获得。

We show that diffusion models can achieve image sample quality superior to the current state-of-the-art generative models. We achieve this on unconditional image synthesis by finding a better architecture through a series of ablations. For conditional image synthesis, we further improve sample quality with classifier guidance: a simple, compute-efficient method for trading off diversity for fidelity using gradients from a classifier. We achieve an FID of 2.97 on ImageNet 128×128, 4.59 on ImageNet 256×256, and 7.72 on ImageNet 512×512, and we match BigGAN-deep even with as few as 25 forward passes per sample, all while maintaining better coverage of the distribution. Finally, we find that classifier guidance combines well with upsampling diffusion models, further improving FID to 3.94 on ImageNet 256×256 and 3.85 on ImageNet 512×512. We release our code at https://github.com/openai/guided-diffusion.

现代监视系统使用基于深度学习的面部验证网络执行人员认可。大多数最先进的面部验证系统都是使用可见光谱图像训练的。但是，在弱光和夜间条件的情况下，在可见光谱中获取图像是不切实际的，并且通常在诸如热红外域之类的替代域中捕获图像。在检索相应的可见域图像后，通常在热图像中进行面部验证。这是一个公认的问题，通常称为热能（T2V）图像翻译。在本文中，我们建议针对面部图像的T2V翻译基于Denoising扩散概率模型（DDPM）解决方案。在训练过程中，该模型通过扩散过程了解了它们相应的热图像，可见面部图像的条件分布。在推断过程中，可见的域图像是通过从高斯噪声开始并反复执行的。 DDPM的现有推理过程是随机且耗时的。因此，我们提出了一种新颖的推理策略，以加快DDPM的推理时间，特别是用于T2V图像翻译问题。我们在多个数据集上实现了最新结果。代码和验证的模型可在http://github.com/nithin-gk/t2v-ddpm上公开获得

In recent years, denoising diffusion models have demonstrated outstanding image generation performance. The information on natural images captured by these models is useful for many image reconstruction applications, where the task is to restore a clean image from its degraded observations. In this work, we propose a conditional sampling scheme that exploits the prior learned by diffusion models while retaining agreement with the observations. We then combine it with a novel approach for adapting pretrained diffusion denoising networks to their input. We examine two adaption strategies: the first uses only the degraded image, while the second, which we advocate, is performed using images that are ``nearest neighbors'' of the degraded image, retrieved from a diverse dataset using an off-the-shelf visual-language model. To evaluate our method, we test it on two state-of-the-art publicly available diffusion models, Stable Diffusion and Guided Diffusion. We show that our proposed `adaptive diffusion for image reconstruction' (ADIR) approach achieves a significant improvement in the super-resolution, deblurring, and text-based editing tasks.

作为生成部件作为自回归模型的向量量化变形式自动化器（VQ-VAE）的集成在图像生成上产生了高质量的结果。但是，自回归模型将严格遵循采样阶段的逐步扫描顺序。这导致现有的VQ系列模型几乎不会逃避缺乏全球信息的陷阱。连续域中的去噪扩散概率模型（DDPM）显示了捕获全局背景的能力，同时产生高质量图像。在离散状态空间中，一些作品已经证明了执行文本生成和低分辨率图像生成的可能性。我们认为，在VQ-VAE的富含内容的离散视觉码本的帮助下，离散扩散模型还可以利用全局上下文产生高保真图像，这补偿了沿像素空间的经典自回归模型的缺陷。同时，离散VAE与扩散模型的集成解决了传统的自回归模型的缺点是超大的，以及在生成图像时需要在采样过程中的过度时间的扩散模型。结果发现所生成的图像的质量严重依赖于离散的视觉码本。广泛的实验表明，所提出的矢量量化离散扩散模型（VQ-DDM）能够实现与低复杂性的顶层方法的相当性能。它还展示了在没有额外培训的图像修复任务方面与自回归模型量化的其他矢量突出的优势。

Generative adversarial networks (GANs) have made great success in image inpainting yet still have difficulties tackling large missing regions. In contrast, iterative algorithms, such as autoregressive and denoising diffusion models, have to be deployed with massive computing resources for decent effect. To overcome the respective limitations, we present a novel spatial diffusion model (SDM) that uses a few iterations to gradually deliver informative pixels to the entire image, largely enhancing the inference efficiency. Also, thanks to the proposed decoupled probabilistic modeling and spatial diffusion scheme, our method achieves high-quality large-hole completion. On multiple benchmarks, we achieve new state-of-the-art performance. Code is released at https://github.com/fenglinglwb/SDM.