Image-based head swapping task aims to stitch a source head to another source body flawlessly. This seldom-studied task faces two major challenges: 1) Preserving the head and body from various sources while generating a seamless transition region. 2) No paired head swapping dataset and benchmark so far. In this paper, we propose an image-based head swapping framework (HS-Diffusion) which consists of a semantic-guided latent diffusion model (SG-LDM) and a semantic layout generator. We blend the semantic layouts of source head and source body, and then inpaint the transition region by the semantic layout generator, achieving a coarse-grained head swapping. SG-LDM can further implement fine-grained head swapping with the blended layout as condition by a progressive fusion process, while preserving source head and source body with high-quality reconstruction. To this end, we design a head-cover augmentation strategy for training and a neck alignment trick for geometric realism. Importantly, we construct a new image-based head swapping benchmark and propose two tailor-designed metrics (Mask-FID and Focal-FID). Extensive experiments demonstrate the superiority of our framework. The code will be available: https://github.com/qinghew/HS-Diffusion.

自由格式介绍是在任意二进制掩码指定的区域中向图像中添加新内容的任务。大多数现有方法训练了一定的面具分布，这将其概括能力限制为看不见的掩模类型。此外，通过像素和知觉损失的训练通常会导致对缺失区域的简单质地扩展，而不是语义上有意义的一代。在这项工作中，我们提出重新启动：基于deno的扩散概率模型（DDPM）的内部介入方法，甚至适用于极端掩模。我们采用预定的无条件DDPM作为生成先验。为了调节生成过程，我们仅通过使用给定的图像信息对未掩盖的区域进行采样来改变反向扩散迭代。由于该技术不会修改或调节原始DDPM网络本身，因此该模型可为任何填充形式产生高质量和不同的输出图像。我们使用标准面具和极端口罩验证面部和通用图像的方法。重新粉刷优于最先进的自动回归，而GAN的方法至少在六个面具分布中进行了五个。 github存储库：git.io/repaint

与生成的对抗网（GAN）相比，降级扩散概率模型（DDPM）在各种图像生成任务中取得了显着成功。关于语义图像综合的最新工作主要遵循\ emph {de exto}基于gan的方法，这可能导致生成图像的质量或多样性不令人满意。在本文中，我们提出了一个基于DDPM的新型框架，用于语义图像合成。与先前的条件扩散模型不同，将语义布局和嘈杂的图像作为输入为U-NET结构，该结构可能无法完全利用输入语义掩码中的信息，我们的框架处理语义布局和嘈杂的图像不同。它将噪声图像馈送到U-NET结构的编码器时，而语义布局通过多层空间自适应归一化操作符将语义布局馈送到解码器。为了进一步提高语义图像合成中的发电质量和语义解释性，我们介绍了无分类器的指导采样策略，该策略承认采样过程的无条件模型的得分。在三个基准数据集上进行的广泛实验证明了我们提出的方法的有效性，从而在忠诚度（FID）和多样性〜（LPIPS）方面实现了最先进的性能。

Structure-guided image completion aims to inpaint a local region of an image according to an input guidance map from users. While such a task enables many practical applications for interactive editing, existing methods often struggle to hallucinate realistic object instances in complex natural scenes. Such a limitation is partially due to the lack of semantic-level constraints inside the hole region as well as the lack of a mechanism to enforce realistic object generation. In this work, we propose a learning paradigm that consists of semantic discriminators and object-level discriminators for improving the generation of complex semantics and objects. Specifically, the semantic discriminators leverage pretrained visual features to improve the realism of the generated visual concepts. Moreover, the object-level discriminators take aligned instances as inputs to enforce the realism of individual objects. Our proposed scheme significantly improves the generation quality and achieves state-of-the-art results on various tasks, including segmentation-guided completion, edge-guided manipulation and panoptically-guided manipulation on Places2 datasets. Furthermore, our trained model is flexible and can support multiple editing use cases, such as object insertion, replacement, removal and standard inpainting. In particular, our trained model combined with a novel automatic image completion pipeline achieves state-of-the-art results on the standard inpainting task.

Generative adversarial networks (GANs) have made great success in image inpainting yet still have difficulties tackling large missing regions. In contrast, iterative algorithms, such as autoregressive and denoising diffusion models, have to be deployed with massive computing resources for decent effect. To overcome the respective limitations, we present a novel spatial diffusion model (SDM) that uses a few iterations to gradually deliver informative pixels to the entire image, largely enhancing the inference efficiency. Also, thanks to the proposed decoupled probabilistic modeling and spatial diffusion scheme, our method achieves high-quality large-hole completion. On multiple benchmarks, we achieve new state-of-the-art performance. Code is released at https://github.com/fenglinglwb/SDM.

可控图像合成模型允许根据文本指令或来自示例图像的指导创建不同的图像。最近，已经显示出去噪扩散概率模型比现有方法产生更现实的图像，并且已在无条件和类条件设置中成功展示。我们探索细粒度，连续控制该模型类，并引入了一种新颖的统一框架，用于语义扩散指导，允许语言或图像指导，或两者。使用图像文本或图像匹配分数的梯度将指导注入预训练的无条件扩散模型中。我们探讨基于剪辑的文本指导，以及以统一形式的基于内容和类型的图像指导。我们的文本引导综合方法可以应用于没有相关文本注释的数据集。我们对FFHQ和LSUN数据集进行实验，并显示出细粒度的文本引导图像合成的结果，与样式或内容示例图像相关的图像的合成，以及具有文本和图像引导的示例。

由于发型的复杂性和美味，编辑发型是独一无二的，而且具有挑战性。尽管最近的方法显着改善了头发的细节，但是当源图像的姿势与目标头发图像的姿势大不相同时，这些模型通常会产生不良的输出，从而限制了其真实世界的应用。发型是一种姿势不变的发型转移模型，可以减轻这种限制，但在保留精致的头发质地方面仍然表现出不令人满意的质量。为了解决这些局限性，我们提出了配备潜在优化和新呈现的局部匹配损失的高性能姿势不变的发型转移模型。在stylegan2潜在空间中，我们首先探索目标头发的姿势对准的潜在代码，并根据本地风格匹配保留了详细纹理。然后，我们的模型对源的遮挡构成了对齐的目标头发的遮挡，并将两个图像混合在一起以产生最终输出。实验结果表明，我们的模型在在较大的姿势差异和保留局部发型纹理下转移发型方面具有优势。

随着信息中的各种方式存在于现实世界中的各种方式，多式联信息之间的有效互动和融合在计算机视觉和深度学习研究中的多模式数据的创造和感知中起着关键作用。通过卓越的功率，在多式联运信息中建模互动，多式联运图像合成和编辑近年来已成为一个热门研究主题。与传统的视觉指导不同，提供明确的线索，多式联路指南在图像合成和编辑方面提供直观和灵活的手段。另一方面，该领域也面临着具有固有的模态差距的特征的几个挑战，高分辨率图像的合成，忠实的评估度量等。在本调查中，我们全面地阐述了最近多式联运图像综合的进展根据数据模型和模型架构编辑和制定分类。我们从图像合成和编辑中的不同类型的引导方式开始介绍。然后，我们描述了多模式图像综合和编辑方法，其具有详细的框架，包括生成的对抗网络（GAN），GaN反转，变压器和其他方法，例如NERF和扩散模型。其次是在多模式图像合成和编辑中广泛采用的基准数据集和相应的评估度量的综合描述，以及分析各个优点和限制的不同合成方法的详细比较。最后，我们为目前的研究挑战和未来的研究方向提供了深入了解。与本调查相关的项目可在HTTPS://github.com/fnzhan/mise上获得

扩散模型（DMS）显示出高质量图像合成的巨大潜力。但是，当涉及到具有复杂场景的图像时，如何正确描述图像全局结构和对象细节仍然是一项具有挑战性的任务。在本文中，我们提出了弗里多（Frido），这是一种特征金字塔扩散模型，该模型执行了图像合成的多尺度粗到1个降解过程。我们的模型将输入图像分解为依赖比例的矢量量化特征，然后是用于产生图像输出的粗到细门。在上述多尺度表示阶段，可以进一步利用文本，场景图或图像布局等其他输入条件。因此，还可以将弗里多应用于条件或跨模式图像合成。我们对各种无条件和有条件的图像生成任务进行了广泛的实验，从文本到图像综合，布局到图像，场景环形图像到标签形象。更具体地说，我们在五个基准测试中获得了最先进的FID分数，即可可和开阔图像的布局到图像，可可和视觉基因组的场景环形图像以及可可的标签对图像图像。 。代码可在https://github.com/davidhalladay/frido上找到。

我们提出了Exe-Gan，这是一种新型的使用生成对抗网络的典范引导的面部介绍框架。我们的方法不仅可以保留输入面部图像的质量，而且还可以使用类似示例性的面部属性来完成图像。我们通过同时利用输入图像的全局样式，从随机潜在代码生成的随机样式以及示例图像的示例样式来实现这一目标。我们介绍了一个新颖的属性相似性指标，以鼓励网络以一种自我监督的方式从示例中学习面部属性的风格。为了确保跨地区边界之间的自然过渡，我们引入了一种新型的空间变体梯度反向传播技术，以根据空间位置调整损耗梯度。关于公共Celeba-HQ和FFHQ数据集的广泛评估和实际应用，可以验证Exe-GAN的优越性，从面部镶嵌的视觉质量来看。

自然语言为图像编辑提供高度直观的界面。在本文中，我们基于自然语言描述与ROI掩模一起介绍用于在通用自然图像中执行局部（基于区域的）编辑的第一解决方案。我们通过利用并结合预先训练的语言图像模型（CLIP）来实现我们的目标，以使编辑朝向用户提供的文本提示，具有去噪扩散概率模型（DDPM）来产生自然的结果。为了使编辑区域与图像的不变部分无缝熔化，我们在噪声水平的进展下使用本地文本引导的扩散潜伏在空间上混合输入图像的声明版本。此外，我们表明向扩散过程增加增强，减轻了对抗性结果。我们与定性和定量的几个基线和相关方法进行比较，并表明我们的方法在整体现实主义方面优于这些解决方案，保留背景和匹配文本的能力。最后，我们显示了多个文本驱动的编辑应用程序，包括将新对象添加到图像，删除/替换/更改现有对象，背景替换和图像外推。

Conditional diffusion probabilistic models can model the distribution of natural images and can generate diverse and realistic samples based on given conditions. However, oftentimes their results can be unrealistic with observable color shifts and textures. We believe that this issue results from the divergence between the probabilistic distribution learned by the model and the distribution of natural images. The delicate conditions gradually enlarge the divergence during each sampling timestep. To address this issue, we introduce a new method that brings the predicted samples to the training data manifold using a pretrained unconditional diffusion model. The unconditional model acts as a regularizer and reduces the divergence introduced by the conditional model at each sampling step. We perform comprehensive experiments to demonstrate the effectiveness of our approach on super-resolution, colorization, turbulence removal, and image-deraining tasks. The improvements obtained by our method suggest that the priors can be incorporated as a general plugin for improving conditional diffusion models.

从手绘中生成图像是内容创建的至关重要和基本任务。翻译很困难，因为存在无限的可能性，并且不同的用户通常会期望不同的结果。因此，我们提出了一个统一的框架，该框架支持基于扩散模型的草图和笔触对图像合成的三维控制。用户不仅可以确定输入笔画和草图的忠诚程度，而且还可以确定现实程度，因为用户输入通常与真实图像不一致。定性和定量实验表明，我们的框架实现了最新的性能，同时提供了具有控制形状，颜色和现实主义的自定义图像的灵活性。此外，我们的方法释放了应用程序，例如在真实图像上编辑，部分草图和笔触的生成以及多域多模式合成。

Generic image inpainting aims to complete a corrupted image by borrowing surrounding information, which barely generates novel content. By contrast, multi-modal inpainting provides more flexible and useful controls on the inpainted content, \eg, a text prompt can be used to describe an object with richer attributes, and a mask can be used to constrain the shape of the inpainted object rather than being only considered as a missing area. We propose a new diffusion-based model named SmartBrush for completing a missing region with an object using both text and shape-guidance. While previous work such as DALLE-2 and Stable Diffusion can do text-guided inapinting they do not support shape guidance and tend to modify background texture surrounding the generated object. Our model incorporates both text and shape guidance with precision control. To preserve the background better, we propose a novel training and sampling strategy by augmenting the diffusion U-net with object-mask prediction. Lastly, we introduce a multi-task training strategy by jointly training inpainting with text-to-image generation to leverage more training data. We conduct extensive experiments showing that our model outperforms all baselines in terms of visual quality, mask controllability, and background preservation.

图像介入寻求一种语义一致的方法，以根据其未掩盖的内容来恢复损坏的图像。以前的方法通常将训练有素的甘恩重复使用，然后在产生逼真的斑块中用于缺少GAN反转的孔。然而，在这些算法中对硬约束的无知可能会产生gan倒置和图像插入之间的差距。在解决这个问题的情况下，我们在本文中设计了一个新颖的GAN反转模型，用于图像插入，称为Interverfill，主要由带有预调制模块的编码器和具有F＆W+潜在空间的GAN生成器组成。在编码器中，预调制网络利用多尺度结构将更多的歧视语义编码为样式向量。为了弥合GAN倒置和图像插入之间的缝隙，提出了F＆W+潜在空间以消除巨大的颜色差异和语义不一致。为了重建忠实和逼真的图像，一个简单而有效的软上升平均潜在模块旨在捕获更多样化的内域模式，以合成大型腐败的高保真质地。在包括Ploce2，Celeba-HQ，Metfaces和Scenery在内的四个具有挑战性的数据集上进行的全面实验表明，我们的Intervill效果优于定性和定量的高级方法，并支持室外图像的完成。

基于图像的虚拟试验是以人为中心的现实潜力，是以人为中心的图像生成的最有希望的应用之一。在这项工作中，我们迈出了一步，探索多功能的虚拟尝试解决方案，我们认为这应该具有三个主要属性，即，它们应支持无监督的培训，任意服装类别和可控的服装编辑。为此，我们提出了一个特征性的端到端网络，即用空间自适应的斑点适应性GAN ++（Pasta-gan ++），以实现用于高分辨率不合规的虚拟试验的多功能系统。具体而言，我们的意大利面++由一个创新的贴布贴片的拆卸模块组成，可以将完整的服装切换为归一化贴剂，该贴片能够保留服装样式信息，同时消除服装空间信息，从而减轻在未受监督训练期间过度适应的问题。此外，面食++引入了基于贴片的服装表示和一个贴片引导的解析合成块，使其可以处理任意服装类别并支持本地服装编辑。最后，为了获得具有逼真的纹理细节的尝试结果，面食gan ++结合了一种新型的空间自适应残留模块，以将粗翘曲的服装功能注入发电机。对我们新收集的未配对的虚拟试验（UPT）数据集进行了广泛的实验，证明了面食gan ++比现有SOTA的优越性及其可控服装编辑的能力。

Object compositing based on 2D images is a challenging problem since it typically involves multiple processing stages such as color harmonization, geometry correction and shadow generation to generate realistic results. Furthermore, annotating training data pairs for compositing requires substantial manual effort from professionals, and is hardly scalable. Thus, with the recent advances in generative models, in this work, we propose a self-supervised framework for object compositing by leveraging the power of conditional diffusion models. Our framework can hollistically address the object compositing task in a unified model, transforming the viewpoint, geometry, color and shadow of the generated object while requiring no manual labeling. To preserve the input object's characteristics, we introduce a content adaptor that helps to maintain categorical semantics and object appearance. A data augmentation method is further adopted to improve the fidelity of the generator. Our method outperforms relevant baselines in both realism and faithfulness of the synthesized result images in a user study on various real-world images.

In this paper, we propose a diffusion-based face swapping framework for the first time, called DiffFace, composed of training ID conditional DDPM, sampling with facial guidance, and a target-preserving blending. In specific, in the training process, the ID conditional DDPM is trained to generate face images with the desired identity. In the sampling process, we use the off-the-shelf facial expert models to make the model transfer source identity while preserving target attributes faithfully. During this process, to preserve the background of the target image and obtain the desired face swapping result, we additionally propose a target-preserving blending strategy. It helps our model to keep the attributes of the target face from noise while transferring the source facial identity. In addition, without any re-training, our model can flexibly apply additional facial guidance and adaptively control the ID-attributes trade-off to achieve the desired results. To the best of our knowledge, this is the first approach that applies the diffusion model in face swapping task. Compared with previous GAN-based approaches, by taking advantage of the diffusion model for the face swapping task, DiffFace achieves better benefits such as training stability, high fidelity, diversity of the samples, and controllability. Extensive experiments show that our DiffFace is comparable or superior to the state-of-the-art methods on several standard face swapping benchmarks.

In this work, we are dedicated to text-guided image generation and propose a novel framework, i.e., CLIP2GAN, by leveraging CLIP model and StyleGAN. The key idea of our CLIP2GAN is to bridge the output feature embedding space of CLIP and the input latent space of StyleGAN, which is realized by introducing a mapping network. In the training stage, we encode an image with CLIP and map the output feature to a latent code, which is further used to reconstruct the image. In this way, the mapping network is optimized in a self-supervised learning way. In the inference stage, since CLIP can embed both image and text into a shared feature embedding space, we replace CLIP image encoder in the training architecture with CLIP text encoder, while keeping the following mapping network as well as StyleGAN model. As a result, we can flexibly input a text description to generate an image. Moreover, by simply adding mapped text features of an attribute to a mapped CLIP image feature, we can effectively edit the attribute to the image. Extensive experiments demonstrate the superior performance of our proposed CLIP2GAN compared to previous methods.

扩散概率模型（DPMS）在竞争对手GANS的图像生成中取得了显着的质量。但与GAN不同，DPMS使用一组缺乏语义含义的一组潜在变量，并且不能作为其他任务的有用表示。本文探讨了使用DPMS进行表示学习的可能性，并寻求通过自动编码提取输入图像的有意义和可解码的表示。我们的主要思想是使用可学习的编码器来发现高级语义，以及DPM作为用于建模剩余随机变化的解码器。我们的方法可以将任何图像编码为两部分潜在的代码，其中第一部分是语义有意义和线性的，第二部分捕获随机细节，允许接近精确的重建。这种功能使当前箔基于GaN的方法的挑战性应用，例如实际图像上的属性操作。我们还表明，这两级编码可提高去噪效率，自然地涉及各种下游任务，包括几次射击条件采样。