与生成的对抗网（GAN）相比，降级扩散概率模型（DDPM）在各种图像生成任务中取得了显着成功。关于语义图像综合的最新工作主要遵循\ emph {de exto}基于gan的方法，这可能导致生成图像的质量或多样性不令人满意。在本文中，我们提出了一个基于DDPM的新型框架，用于语义图像合成。与先前的条件扩散模型不同，将语义布局和嘈杂的图像作为输入为U-NET结构，该结构可能无法完全利用输入语义掩码中的信息，我们的框架处理语义布局和嘈杂的图像不同。它将噪声图像馈送到U-NET结构的编码器时，而语义布局通过多层空间自适应归一化操作符将语义布局馈送到解码器。为了进一步提高语义图像合成中的发电质量和语义解释性，我们介绍了无分类器的指导采样策略，该策略承认采样过程的无条件模型的得分。在三个基准数据集上进行的广泛实验证明了我们提出的方法的有效性，从而在忠诚度（FID）和多样性〜（LPIPS）方面实现了最先进的性能。

可控图像合成模型允许根据文本指令或来自示例图像的指导创建不同的图像。最近，已经显示出去噪扩散概率模型比现有方法产生更现实的图像，并且已在无条件和类条件设置中成功展示。我们探索细粒度，连续控制该模型类，并引入了一种新颖的统一框架，用于语义扩散指导，允许语言或图像指导，或两者。使用图像文本或图像匹配分数的梯度将指导注入预训练的无条件扩散模型中。我们探讨基于剪辑的文本指导，以及以统一形式的基于内容和类型的图像指导。我们的文本引导综合方法可以应用于没有相关文本注释的数据集。我们对FFHQ和LSUN数据集进行实验，并显示出细粒度的文本引导图像合成的结果，与样式或内容示例图像相关的图像的合成，以及具有文本和图像引导的示例。

数字艺术合成在多媒体社区中受到越来越多的关注，因为有效地与公众参与了艺术。当前的数字艺术合成方法通常使用单模式输入作为指导，从而限制了模型的表现力和生成结果的多样性。为了解决这个问题，我们提出了多模式引导的艺术品扩散（MGAD）模型，该模型是一种基于扩散的数字艺术品生成方法，它利用多模式提示作为控制无分类器扩散模型的指导。此外，对比度语言图像预处理（剪辑）模型用于统一文本和图像模式。关于生成的数字艺术绘画质量和数量的广泛实验结果证实了扩散模型和多模式指导的组合有效性。代码可从https://github.com/haha-lisa/mgad-multimodal-guided-artwork-diffusion获得。

自由格式介绍是在任意二进制掩码指定的区域中向图像中添加新内容的任务。大多数现有方法训练了一定的面具分布，这将其概括能力限制为看不见的掩模类型。此外，通过像素和知觉损失的训练通常会导致对缺失区域的简单质地扩展，而不是语义上有意义的一代。在这项工作中，我们提出重新启动：基于deno的扩散概率模型（DDPM）的内部介入方法，甚至适用于极端掩模。我们采用预定的无条件DDPM作为生成先验。为了调节生成过程，我们仅通过使用给定的图像信息对未掩盖的区域进行采样来改变反向扩散迭代。由于该技术不会修改或调节原始DDPM网络本身，因此该模型可为任何填充形式产生高质量和不同的输出图像。我们使用标准面具和极端口罩验证面部和通用图像的方法。重新粉刷优于最先进的自动回归，而GAN的方法至少在六个面具分布中进行了五个。 github存储库：git.io/repaint

Conditional diffusion probabilistic models can model the distribution of natural images and can generate diverse and realistic samples based on given conditions. However, oftentimes their results can be unrealistic with observable color shifts and textures. We believe that this issue results from the divergence between the probabilistic distribution learned by the model and the distribution of natural images. The delicate conditions gradually enlarge the divergence during each sampling timestep. To address this issue, we introduce a new method that brings the predicted samples to the training data manifold using a pretrained unconditional diffusion model. The unconditional model acts as a regularizer and reduces the divergence introduced by the conditional model at each sampling step. We perform comprehensive experiments to demonstrate the effectiveness of our approach on super-resolution, colorization, turbulence removal, and image-deraining tasks. The improvements obtained by our method suggest that the priors can be incorporated as a general plugin for improving conditional diffusion models.

随着信息中的各种方式存在于现实世界中的各种方式，多式联信息之间的有效互动和融合在计算机视觉和深度学习研究中的多模式数据的创造和感知中起着关键作用。通过卓越的功率，在多式联运信息中建模互动，多式联运图像合成和编辑近年来已成为一个热门研究主题。与传统的视觉指导不同，提供明确的线索，多式联路指南在图像合成和编辑方面提供直观和灵活的手段。另一方面，该领域也面临着具有固有的模态差距的特征的几个挑战，高分辨率图像的合成，忠实的评估度量等。在本调查中，我们全面地阐述了最近多式联运图像综合的进展根据数据模型和模型架构编辑和制定分类。我们从图像合成和编辑中的不同类型的引导方式开始介绍。然后，我们描述了多模式图像综合和编辑方法，其具有详细的框架，包括生成的对抗网络（GAN），GaN反转，变压器和其他方法，例如NERF和扩散模型。其次是在多模式图像合成和编辑中广泛采用的基准数据集和相应的评估度量的综合描述，以及分析各个优点和限制的不同合成方法的详细比较。最后，我们为目前的研究挑战和未来的研究方向提供了深入了解。与本调查相关的项目可在HTTPS://github.com/fnzhan/mise上获得

Generating photos satisfying multiple constraints find broad utility in the content creation industry. A key hurdle to accomplishing this task is the need for paired data consisting of all modalities (i.e., constraints) and their corresponding output. Moreover, existing methods need retraining using paired data across all modalities to introduce a new condition. This paper proposes a solution to this problem based on denoising diffusion probabilistic models (DDPMs). Our motivation for choosing diffusion models over other generative models comes from the flexible internal structure of diffusion models. Since each sampling step in the DDPM follows a Gaussian distribution, we show that there exists a closed-form solution for generating an image given various constraints. Our method can unite multiple diffusion models trained on multiple sub-tasks and conquer the combined task through our proposed sampling strategy. We also introduce a novel reliability parameter that allows using different off-the-shelf diffusion models trained across various datasets during sampling time alone to guide it to the desired outcome satisfying multiple constraints. We perform experiments on various standard multimodal tasks to demonstrate the effectiveness of our approach. More details can be found in https://nithin-gk.github.io/projectpages/Multidiff/index.html

Generative adversarial networks (GANs) have made great success in image inpainting yet still have difficulties tackling large missing regions. In contrast, iterative algorithms, such as autoregressive and denoising diffusion models, have to be deployed with massive computing resources for decent effect. To overcome the respective limitations, we present a novel spatial diffusion model (SDM) that uses a few iterations to gradually deliver informative pixels to the entire image, largely enhancing the inference efficiency. Also, thanks to the proposed decoupled probabilistic modeling and spatial diffusion scheme, our method achieves high-quality large-hole completion. On multiple benchmarks, we achieve new state-of-the-art performance. Code is released at https://github.com/fenglinglwb/SDM.

扩散模型（DMS）显示出高质量图像合成的巨大潜力。但是，当涉及到具有复杂场景的图像时，如何正确描述图像全局结构和对象细节仍然是一项具有挑战性的任务。在本文中，我们提出了弗里多（Frido），这是一种特征金字塔扩散模型，该模型执行了图像合成的多尺度粗到1个降解过程。我们的模型将输入图像分解为依赖比例的矢量量化特征，然后是用于产生图像输出的粗到细门。在上述多尺度表示阶段，可以进一步利用文本，场景图或图像布局等其他输入条件。因此，还可以将弗里多应用于条件或跨模式图像合成。我们对各种无条件和有条件的图像生成任务进行了广泛的实验，从文本到图像综合，布局到图像，场景环形图像到标签形象。更具体地说，我们在五个基准测试中获得了最先进的FID分数，即可可和开阔图像的布局到图像，可可和视觉基因组的场景环形图像以及可可的标签对图像图像。 。代码可在https://github.com/davidhalladay/frido上找到。

从手绘中生成图像是内容创建的至关重要和基本任务。翻译很困难，因为存在无限的可能性，并且不同的用户通常会期望不同的结果。因此，我们提出了一个统一的框架，该框架支持基于扩散模型的草图和笔触对图像合成的三维控制。用户不仅可以确定输入笔画和草图的忠诚程度，而且还可以确定现实程度，因为用户输入通常与真实图像不一致。定性和定量实验表明，我们的框架实现了最新的性能，同时提供了具有控制形状，颜色和现实主义的自定义图像的灵活性。此外，我们的方法释放了应用程序，例如在真实图像上编辑，部分草图和笔触的生成以及多域多模式合成。

DeNoising扩散模型代表了计算机视觉中最新的主题，在生成建模领域表现出了显着的结果。扩散模型是一个基于两个阶段的深层生成模型，一个正向扩散阶段和反向扩散阶段。在正向扩散阶段，通过添加高斯噪声，输入数据在几个步骤中逐渐受到干扰。在反向阶段，模型的任务是通过学习逐步逆转扩散过程来恢复原始输入数据。尽管已知的计算负担，即由于采样过程中涉及的步骤数量，扩散模型对生成样品的质量和多样性得到了广泛赞赏。在这项调查中，我们对视觉中应用的denoising扩散模型的文章进行了全面综述，包括该领域的理论和实际贡献。首先，我们识别并介绍了三个通用扩散建模框架，这些框架基于扩散概率模型，噪声调节得分网络和随机微分方程。我们进一步讨论了扩散模型与其他深层生成模型之间的关系，包括变异自动编码器，生成对抗网络，基于能量的模型，自回归模型和正常流量。然后，我们介绍了计算机视觉中应用的扩散模型的多角度分类。最后，我们说明了扩散模型的当前局限性，并设想了一些有趣的未来研究方向。

我们提出了一种基于示例的图像翻译的新方法，称为匹配交织的扩散模型（MIDMS）。该任务的大多数现有方法都是基于GAN的匹配，然后代表了代代框架。但是，在此框架中，跨跨域的语义匹配难度引起的匹配误差，例如草图和照片，可以很容易地传播到生成步骤，从而导致结果退化。由于扩散模型的最新成功激发了克服GAN的缺点，我们结合了扩散模型以克服这些局限性。具体而言，我们制定了一个基于扩散的匹配和生成框架，该框架通过将中间扭曲馈入尖锐的过程并将其变形以生成翻译的图像，从而交织了潜在空间中的跨域匹配和扩散步骤。此外，为了提高扩散过程的可靠性，我们使用周期一致性设计了一种置信度的过程，以在翻译过程中仅考虑自信区域。实验结果表明，我们的MIDM比最新方法产生的图像更合理。

In recent years, generative models have undergone significant advancement due to the success of diffusion models. The success of these models is often attributed to their use of guidance techniques, such as classifier and classifier-free methods, which provides effective mechanisms to trade-off between fidelity and diversity. However, these methods are not capable of guiding a generated image to be aware of its geometric configuration, e.g., depth, which hinders the application of diffusion models to areas that require a certain level of depth awareness. To address this limitation, we propose a novel guidance approach for diffusion models that uses estimated depth information derived from the rich intermediate representations of diffusion models. To do this, we first present a label-efficient depth estimation framework using the internal representations of diffusion models. At the sampling phase, we utilize two guidance techniques to self-condition the generated image using the estimated depth map, the first of which uses pseudo-labeling, and the subsequent one uses a depth-domain diffusion prior. Experiments and extensive ablation studies demonstrate the effectiveness of our method in guiding the diffusion models toward geometrically plausible image generation. Project page is available at https://ku-cvlab.github.io/DAG/.

作为生成部件作为自回归模型的向量量化变形式自动化器（VQ-VAE）的集成在图像生成上产生了高质量的结果。但是，自回归模型将严格遵循采样阶段的逐步扫描顺序。这导致现有的VQ系列模型几乎不会逃避缺乏全球信息的陷阱。连续域中的去噪扩散概率模型（DDPM）显示了捕获全局背景的能力，同时产生高质量图像。在离散状态空间中，一些作品已经证明了执行文本生成和低分辨率图像生成的可能性。我们认为，在VQ-VAE的富含内容的离散视觉码本的帮助下，离散扩散模型还可以利用全局上下文产生高保真图像，这补偿了沿像素空间的经典自回归模型的缺陷。同时，离散VAE与扩散模型的集成解决了传统的自回归模型的缺点是超大的，以及在生成图像时需要在采样过程中的过度时间的扩散模型。结果发现所生成的图像的质量严重依赖于离散的视觉码本。广泛的实验表明，所提出的矢量量化离散扩散模型（VQ-DDM）能够实现与低复杂性的顶层方法的相当性能。它还展示了在没有额外培训的图像修复任务方面与自回归模型量化的其他矢量突出的优势。

Diffusion models are rising as a powerful solution for high-fidelity image generation, which exceeds GANs in quality in many circumstances. However, their slow training and inference speed is a huge bottleneck, blocking them from being used in real-time applications. A recent DiffusionGAN method significantly decreases the models' running time by reducing the number of sampling steps from thousands to several, but their speeds still largely lag behind the GAN counterparts. This paper aims to reduce the speed gap by proposing a novel wavelet-based diffusion structure. We extract low-and-high frequency components from both image and feature levels via wavelet decomposition and adaptively handle these components for faster processing while maintaining good generation quality. Furthermore, we propose to use a reconstruction term, which effectively boosts the model training convergence. Experimental results on CelebA-HQ, CIFAR-10, LSUN-Church, and STL-10 datasets prove our solution is a stepping-stone to offering real-time and high-fidelity diffusion models. Our code and pre-trained checkpoints will be available at \url{https://github.com/VinAIResearch/WaveDiff.git}.

denoisis扩散概率模型（DDPM）能够通过引入独立的噪声吸引分类器来在每次deosoing过程的时间步骤中提供条件梯度指导，从而使有条件的图像从先前的噪声到真实数据。但是，由于分类器能够轻松地区分不完全生成的图像仅具有高级结构的能力，因此梯度是一种类信息指导，倾向于尽早消失，导致从条件生成过程中崩溃到无条件过程。为了解决这个问题，我们从两个角度提出了两种简单但有效的方法。对于抽样程序，我们将预测分布的熵作为指导消失水平的度量，并提出一种熵感知的缩放方法，以适应性地恢复条件语义指导。每个生成样品的％。对于训练阶段，我们提出了熵吸引的优化目标，以减轻噪音数据的过度自信预测。在Imagenet1000 256x256中，我们提出的采样方案和训练有素的分类器（预训练的条件和无条件的DDPM模型可以实现10.89％（4.59至4.59至4.09））和43.5％（12至6.78）FID改善。

In this work, we are dedicated to text-guided image generation and propose a novel framework, i.e., CLIP2GAN, by leveraging CLIP model and StyleGAN. The key idea of our CLIP2GAN is to bridge the output feature embedding space of CLIP and the input latent space of StyleGAN, which is realized by introducing a mapping network. In the training stage, we encode an image with CLIP and map the output feature to a latent code, which is further used to reconstruct the image. In this way, the mapping network is optimized in a self-supervised learning way. In the inference stage, since CLIP can embed both image and text into a shared feature embedding space, we replace CLIP image encoder in the training architecture with CLIP text encoder, while keeping the following mapping network as well as StyleGAN model. As a result, we can flexibly input a text description to generate an image. Moreover, by simply adding mapped text features of an attribute to a mapped CLIP image feature, we can effectively edit the attribute to the image. Extensive experiments demonstrate the superior performance of our proposed CLIP2GAN compared to previous methods.

In this paper, we propose a diffusion-based face swapping framework for the first time, called DiffFace, composed of training ID conditional DDPM, sampling with facial guidance, and a target-preserving blending. In specific, in the training process, the ID conditional DDPM is trained to generate face images with the desired identity. In the sampling process, we use the off-the-shelf facial expert models to make the model transfer source identity while preserving target attributes faithfully. During this process, to preserve the background of the target image and obtain the desired face swapping result, we additionally propose a target-preserving blending strategy. It helps our model to keep the attributes of the target face from noise while transferring the source facial identity. In addition, without any re-training, our model can flexibly apply additional facial guidance and adaptively control the ID-attributes trade-off to achieve the desired results. To the best of our knowledge, this is the first approach that applies the diffusion model in face swapping task. Compared with previous GAN-based approaches, by taking advantage of the diffusion model for the face swapping task, DiffFace achieves better benefits such as training stability, high fidelity, diversity of the samples, and controllability. Extensive experiments show that our DiffFace is comparable or superior to the state-of-the-art methods on several standard face swapping benchmarks.

Image-based head swapping task aims to stitch a source head to another source body flawlessly. This seldom-studied task faces two major challenges: 1) Preserving the head and body from various sources while generating a seamless transition region. 2) No paired head swapping dataset and benchmark so far. In this paper, we propose an image-based head swapping framework (HS-Diffusion) which consists of a semantic-guided latent diffusion model (SG-LDM) and a semantic layout generator. We blend the semantic layouts of source head and source body, and then inpaint the transition region by the semantic layout generator, achieving a coarse-grained head swapping. SG-LDM can further implement fine-grained head swapping with the blended layout as condition by a progressive fusion process, while preserving source head and source body with high-quality reconstruction. To this end, we design a head-cover augmentation strategy for training and a neck alignment trick for geometric realism. Importantly, we construct a new image-based head swapping benchmark and propose two tailor-designed metrics (Mask-FID and Focal-FID). Extensive experiments demonstrate the superiority of our framework. The code will be available: https://github.com/qinghew/HS-Diffusion.