Denoising diffusion models (DDMs) have led to staggering performance leaps in image generation, editing and restoration. However, existing DDMs use very large datasets for training. Here, we introduce a framework for training a DDM on a single image. Our method, which we coin SinDDM, learns the internal statistics of the training image by using a multi-scale diffusion process. To drive the reverse diffusion process, we use a fully-convolutional light-weight denoiser, which is conditioned on both the noise level and the scale. This architecture allows generating samples of arbitrary dimensions, in a coarse-to-fine manner. As we illustrate, SinDDM generates diverse high-quality samples, and is applicable in a wide array of tasks, including style transfer and harmonization. Furthermore, it can be easily guided by external supervision. Particularly, we demonstrate text-guided generation from a single image using a pre-trained CLIP model.

Recent works on diffusion models have demonstrated a strong capability for conditioning image generation, e.g., text-guided image synthesis. Such success inspires many efforts trying to use large-scale pre-trained diffusion models for tackling a challenging problem--real image editing. Works conducted in this area learn a unique textual token corresponding to several images containing the same object. However, under many circumstances, only one image is available, such as the painting of the Girl with a Pearl Earring. Using existing works on fine-tuning the pre-trained diffusion models with a single image causes severe overfitting issues. The information leakage from the pre-trained diffusion models makes editing can not keep the same content as the given image while creating new features depicted by the language guidance. This work aims to address the problem of single-image editing. We propose a novel model-based guidance built upon the classifier-free guidance so that the knowledge from the model trained on a single image can be distilled into the pre-trained diffusion model, enabling content creation even with one given image. Additionally, we propose a patch-based fine-tuning that can effectively help the model generate images of arbitrary resolution. We provide extensive experiments to validate the design choices of our approach and show promising editing capabilities, including changing style, content addition, and object manipulation. The code is available for research purposes at https://github.com/zhang-zx/SINE.git .

Random samples from a single image Single training image Figure 1: Image generation learned from a single training image. We propose SinGAN-a new unconditional generative model trained on a single natural image. Our model learns the image's patch statistics across multiple scales, using a dedicated multi-scale adversarial training scheme; it can then be used to generate new realistic image samples that preserve the original patch distribution while creating new object configurations and structures.

数字艺术合成在多媒体社区中受到越来越多的关注，因为有效地与公众参与了艺术。当前的数字艺术合成方法通常使用单模式输入作为指导，从而限制了模型的表现力和生成结果的多样性。为了解决这个问题，我们提出了多模式引导的艺术品扩散（MGAD）模型，该模型是一种基于扩散的数字艺术品生成方法，它利用多模式提示作为控制无分类器扩散模型的指导。此外，对比度语言图像预处理（剪辑）模型用于统一文本和图像模式。关于生成的数字艺术绘画质量和数量的广泛实验结果证实了扩散模型和多模式指导的组合有效性。代码可从https://github.com/haha-lisa/mgad-multimodal-guided-artwork-diffusion获得。

自然语言为图像编辑提供高度直观的界面。在本文中，我们基于自然语言描述与ROI掩模一起介绍用于在通用自然图像中执行局部（基于区域的）编辑的第一解决方案。我们通过利用并结合预先训练的语言图像模型（CLIP）来实现我们的目标，以使编辑朝向用户提供的文本提示，具有去噪扩散概率模型（DDPM）来产生自然的结果。为了使编辑区域与图像的不变部分无缝熔化，我们在噪声水平的进展下使用本地文本引导的扩散潜伏在空间上混合输入图像的声明版本。此外，我们表明向扩散过程增加增强，减轻了对抗性结果。我们与定性和定量的几个基线和相关方法进行比较，并表明我们的方法在整体现实主义方面优于这些解决方案，保留背景和匹配文本的能力。最后，我们显示了多个文本驱动的编辑应用程序，包括将新对象添加到图像，删除/替换/更改现有对象，背景替换和图像外推。

可控图像合成模型允许根据文本指令或来自示例图像的指导创建不同的图像。最近，已经显示出去噪扩散概率模型比现有方法产生更现实的图像，并且已在无条件和类条件设置中成功展示。我们探索细粒度，连续控制该模型类，并引入了一种新颖的统一框架，用于语义扩散指导，允许语言或图像指导，或两者。使用图像文本或图像匹配分数的梯度将指导注入预训练的无条件扩散模型中。我们探讨基于剪辑的文本指导，以及以统一形式的基于内容和类型的图像指导。我们的文本引导综合方法可以应用于没有相关文本注释的数据集。我们对FFHQ和LSUN数据集进行实验，并显示出细粒度的文本引导图像合成的结果，与样式或内容示例图像相关的图像的合成，以及具有文本和图像引导的示例。

最近，大规模文本驱动的合成模型由于其出色的产生高度多样化的图像而引起了很多关注，这些图像遵循给定的文本提示。这种基于文本的综合方法特别有吸引力，这些方法对人类用来口头描述其意图。因此，将文本驱动的图像合成扩展到文本驱动的图像编辑是很自然的。编辑对于这些生成模型来说是具有挑战性的，因为编辑技术的先天属性是保留大多数原始图像，而在基于文本的模型中，即使对文本提示的小修改也通常会导致完全不同的结果。最先进的方法可以通过要求用户提供空间掩码来本地化编辑，从而忽略蒙版区域内的原始结构和内容，从而减轻这种方法。在本文中，我们追求一个直观的及时提示编辑框架，其中编辑仅由文本控制。为此，我们深入分析了一个文本条件模型，并观察到跨注意层是控制图像的空间布局与提示中每个单词之间关系的关键。通过此观察，我们提出了几种应用程序，它们仅通过编辑文本提示来监视图像综合。这包括通过替换单词，通过添加规范来替换单词编辑的本地化编辑，甚至精心控制单词在图像中反映的程度。我们介绍了各种图像和提示的结果，证明了对编辑提示的高质量综合和忠诚度。

Large-scale diffusion-based generative models have led to breakthroughs in text-conditioned high-resolution image synthesis. Starting from random noise, such text-to-image diffusion models gradually synthesize images in an iterative fashion while conditioning on text prompts. We find that their synthesis behavior qualitatively changes throughout this process: Early in sampling, generation strongly relies on the text prompt to generate text-aligned content, while later, the text conditioning is almost entirely ignored. This suggests that sharing model parameters throughout the entire generation process may not be ideal. Therefore, in contrast to existing works, we propose to train an ensemble of text-to-image diffusion models specialized for different synthesis stages. To maintain training efficiency, we initially train a single model, which is then split into specialized models that are trained for the specific stages of the iterative generation process. Our ensemble of diffusion models, called eDiff-I, results in improved text alignment while maintaining the same inference computation cost and preserving high visual quality, outperforming previous large-scale text-to-image diffusion models on the standard benchmark. In addition, we train our model to exploit a variety of embeddings for conditioning, including the T5 text, CLIP text, and CLIP image embeddings. We show that these different embeddings lead to different behaviors. Notably, the CLIP image embedding allows an intuitive way of transferring the style of a reference image to the target text-to-image output. Lastly, we show a technique that enables eDiff-I's "paint-with-words" capability. A user can select the word in the input text and paint it in a canvas to control the output, which is very handy for crafting the desired image in mind. The project page is available at https://deepimagination.cc/eDiff-I/

We introduce M-VADER: a diffusion model (DM) for image generation where the output can be specified using arbitrary combinations of images and text. We show how M-VADER enables the generation of images specified using combinations of image and text, and combinations of multiple images. Previously, a number of successful DM image generation algorithms have been introduced that make it possible to specify the output image using a text prompt. Inspired by the success of those models, and led by the notion that language was already developed to describe the elements of visual contexts that humans find most important, we introduce an embedding model closely related to a vision-language model. Specifically, we introduce the embedding model S-MAGMA: a 13 billion parameter multimodal decoder combining components from an autoregressive vision-language model MAGMA and biases finetuned for semantic search.

Generic image inpainting aims to complete a corrupted image by borrowing surrounding information, which barely generates novel content. By contrast, multi-modal inpainting provides more flexible and useful controls on the inpainted content, \eg, a text prompt can be used to describe an object with richer attributes, and a mask can be used to constrain the shape of the inpainted object rather than being only considered as a missing area. We propose a new diffusion-based model named SmartBrush for completing a missing region with an object using both text and shape-guidance. While previous work such as DALLE-2 and Stable Diffusion can do text-guided inapinting they do not support shape guidance and tend to modify background texture surrounding the generated object. Our model incorporates both text and shape guidance with precision control. To preserve the background better, we propose a novel training and sampling strategy by augmenting the diffusion U-net with object-mask prediction. Lastly, we introduce a multi-task training strategy by jointly training inpainting with text-to-image generation to leverage more training data. We conduct extensive experiments showing that our model outperforms all baselines in terms of visual quality, mask controllability, and background preservation.

最近已被证明扩散模型产生高质量的合成图像，尤其是与指导技术配对，以促进忠诚的多样性。我们探索文本条件图像综合问题的扩散模型，并比较了两种不同的指导策略：剪辑指导和自由分类指导。我们发现后者是人类评估者的优选，用于光敏和标题相似度，并且通常产生光素质拟种样品。使用自由分类指导的35亿参数文本条件扩散模型的样本由人类评估者对来自Dall-E的人的人们青睐，即使后者使用昂贵的剪辑重新划分。此外，我们发现我们的模型可以进行微调，以执行图像修复，从而实现强大的文本驱动的图像编辑。我们在过滤的数据集中培训较小的模型，并在https://github.com/openai/glide-text2im释放代码和权重。

Generative models, particularly GANs, have been utilized for image editing. Although GAN-based methods perform well on generating reasonable contents aligned with the user's intentions, they struggle to strictly preserve the contents outside the editing region. To address this issue, we use diffusion models instead of GANs and propose a novel image-editing method, based on pixel-wise guidance. Specifically, we first train pixel-classifiers with few annotated data and then estimate the semantic segmentation map of a target image. Users then manipulate the map to instruct how the image is to be edited. The diffusion model generates an edited image via guidance by pixel-wise classifiers, such that the resultant image aligns with the manipulated map. As the guidance is conducted pixel-wise, the proposed method can create reasonable contents in the editing region while preserving the contents outside this region. The experimental results validate the advantages of the proposed method both quantitatively and qualitatively.

Large, text-conditioned generative diffusion models have recently gained a lot of attention for their impressive performance in generating high-fidelity images from text alone. However, achieving high-quality results is almost unfeasible in a one-shot fashion. On the contrary, text-guided image generation involves the user making many slight changes to inputs in order to iteratively carve out the envisioned image. However, slight changes to the input prompt often lead to entirely different images being generated, and thus the control of the artist is limited in its granularity. To provide flexibility, we present the Stable Artist, an image editing approach enabling fine-grained control of the image generation process. The main component is semantic guidance (SEGA) which steers the diffusion process along variable numbers of semantic directions. This allows for subtle edits to images, changes in composition and style, as well as optimization of the overall artistic conception. Furthermore, SEGA enables probing of latent spaces to gain insights into the representation of concepts learned by the model, even complex ones such as 'carbon emission'. We demonstrate the Stable Artist on several tasks, showcasing high-quality image editing and composition.

随着信息中的各种方式存在于现实世界中的各种方式，多式联信息之间的有效互动和融合在计算机视觉和深度学习研究中的多模式数据的创造和感知中起着关键作用。通过卓越的功率，在多式联运信息中建模互动，多式联运图像合成和编辑近年来已成为一个热门研究主题。与传统的视觉指导不同，提供明确的线索，多式联路指南在图像合成和编辑方面提供直观和灵活的手段。另一方面，该领域也面临着具有固有的模态差距的特征的几个挑战，高分辨率图像的合成，忠实的评估度量等。在本调查中，我们全面地阐述了最近多式联运图像综合的进展根据数据模型和模型架构编辑和制定分类。我们从图像合成和编辑中的不同类型的引导方式开始介绍。然后，我们描述了多模式图像综合和编辑方法，其具有详细的框架，包括生成的对抗网络（GAN），GaN反转，变压器和其他方法，例如NERF和扩散模型。其次是在多模式图像合成和编辑中广泛采用的基准数据集和相应的评估度量的综合描述，以及分析各个优点和限制的不同合成方法的详细比较。最后，我们为目前的研究挑战和未来的研究方向提供了深入了解。与本调查相关的项目可在HTTPS://github.com/fnzhan/mise上获得

Considerable progress has recently been made in leveraging CLIP (Contrastive Language-Image Pre-Training) models for text-guided image manipulation. However, all existing works rely on additional generative models to ensure the quality of results, because CLIP alone cannot provide enough guidance information for fine-scale pixel-level changes. In this paper, we introduce CLIPVG, a text-guided image manipulation framework using differentiable vector graphics, which is also the first CLIP-based general image manipulation framework that does not require any additional generative models. We demonstrate that CLIPVG can not only achieve state-of-art performance in both semantic correctness and synthesis quality, but also is flexible enough to support various applications far beyond the capability of all existing methods.

DeNoising扩散模型代表了计算机视觉中最新的主题，在生成建模领域表现出了显着的结果。扩散模型是一个基于两个阶段的深层生成模型，一个正向扩散阶段和反向扩散阶段。在正向扩散阶段，通过添加高斯噪声，输入数据在几个步骤中逐渐受到干扰。在反向阶段，模型的任务是通过学习逐步逆转扩散过程来恢复原始输入数据。尽管已知的计算负担，即由于采样过程中涉及的步骤数量，扩散模型对生成样品的质量和多样性得到了广泛赞赏。在这项调查中，我们对视觉中应用的denoising扩散模型的文章进行了全面综述，包括该领域的理论和实际贡献。首先，我们识别并介绍了三个通用扩散建模框架，这些框架基于扩散概率模型，噪声调节得分网络和随机微分方程。我们进一步讨论了扩散模型与其他深层生成模型之间的关系，包括变异自动编码器，生成对抗网络，基于能量的模型，自回归模型和正常流量。然后，我们介绍了计算机视觉中应用的扩散模型的多角度分类。最后，我们说明了扩散模型的当前局限性，并设想了一些有趣的未来研究方向。

最近，GaN反演方法与对比语言 - 图像预先绘制（CLIP）相结合，可以通过文本提示引导零拍摄图像操作。然而，由于GaN反转能力有限，它们对不同实物的不同实物的应用仍然困难。具体地，这些方法通常在与训练数据相比，改变对象标识或产生不需要的图像伪影的比较与新颖姿势，视图和高度可变内容重建具有新颖姿势，视图和高度可变内容的困难。为了减轻这些问题并实现真实图像的忠实操纵，我们提出了一种新的方法，Dumbused Clip，其使用扩散模型执行文本驱动的图像操纵。基于近期扩散模型的完整反转能力和高质量的图像生成功率，即使在看不见的域之间也成功地执行零拍摄图像操作。此外，我们提出了一种新颖的噪声组合方法，允许简单的多属性操作。与现有基线相比，广泛的实验和人类评估确认了我们的方法的稳健和卓越的操纵性能。

In recent years, denoising diffusion models have demonstrated outstanding image generation performance. The information on natural images captured by these models is useful for many image reconstruction applications, where the task is to restore a clean image from its degraded observations. In this work, we propose a conditional sampling scheme that exploits the prior learned by diffusion models while retaining agreement with the observations. We then combine it with a novel approach for adapting pretrained diffusion denoising networks to their input. We examine two adaption strategies: the first uses only the degraded image, while the second, which we advocate, is performed using images that are ``nearest neighbors'' of the degraded image, retrieved from a diverse dataset using an off-the-shelf visual-language model. To evaluate our method, we test it on two state-of-the-art publicly available diffusion models, Stable Diffusion and Guided Diffusion. We show that our proposed `adaptive diffusion for image reconstruction' (ADIR) approach achieves a significant improvement in the super-resolution, deblurring, and text-based editing tasks.

从手绘中生成图像是内容创建的至关重要和基本任务。翻译很困难，因为存在无限的可能性，并且不同的用户通常会期望不同的结果。因此，我们提出了一个统一的框架，该框架支持基于扩散模型的草图和笔触对图像合成的三维控制。用户不仅可以确定输入笔画和草图的忠诚程度，而且还可以确定现实程度，因为用户输入通常与真实图像不一致。定性和定量实验表明，我们的框架实现了最新的性能，同时提供了具有控制形状，颜色和现实主义的自定义图像的灵活性。此外，我们的方法释放了应用程序，例如在真实图像上编辑，部分草图和笔触的生成以及多域多模式合成。

Text-guided diffusion models have shown superior performance in image/video generation and editing. While few explorations have been performed in 3D scenarios. In this paper, we discuss three fundamental and interesting problems on this topic. First, we equip text-guided diffusion models to achieve $\textbf{3D-consistent generation}$. Specifically, we integrate a NeRF-like neural field to generate low-resolution coarse results for a given camera view. Such results can provide 3D priors as condition information for the following diffusion process. During denoising diffusion, we further enhance the 3D consistency by modeling cross-view correspondences with a novel two-stream (corresponding to two different views) asynchronous diffusion process. Second, we study $\textbf{3D local editing}$ and propose a two-step solution that can generate 360$^{\circ}$ manipulated results by editing an object from a single view. Step 1, we propose to perform 2D local editing by blending the predicted noises. Step 2, we conduct a noise-to-text inversion process that maps 2D blended noises into the view-independent text embedding space. Once the corresponding text embedding is obtained, 360$^{\circ}$ images can be generated. Last but not least, we extend our model to perform \textbf{one-shot novel view synthesis} by fine-tuning on a single image, firstly showing the potential of leveraging text guidance for novel view synthesis. Extensive experiments and various applications show the prowess of our 3DDesigner. The project page is available at https://3ddesigner-diffusion.github.io/.