Learning a good image prior is a long-term goal for image restoration and manipulation. While existing methods like deep image prior (DIP) capture low-level image statistics, there are still gaps toward an image prior that captures rich image semantics including color, spatial coherence, textures, and high-level concepts. This work presents an effective way to exploit the image prior captured by a generative adversarial network (GAN) trained on large-scale natural images. As shown in Fig. 1, the deep generative prior (DGP) provides compelling results to restore missing semantics, e.g., color, patch, resolution, of various degraded images. It also enables diverse image manipulation including random jittering, image morphing, and category transfer. Such highly flexible restoration and manipulation are made possible through relaxing the assumption of existing GAN-inversion methods, which tend to fix the generator. Notably, we allow the generator to be fine-tuned on-the-fly in a progressive manner regularized by feature distance obtained by the discriminator in GAN. We show that these easy-to-implement and practical changes help preserve the reconstruction to remain in the manifold of nature image, and thus lead to more precise and faithful reconstruction for real images. Code is available at https://github.com/XingangPan/deepgenerative-prior.

edu.hk (a) Image Reconstruction (b) Image Colorization (c) Image Super-Resolution (d) Image Denoising (e) Image Inpainting (f) Semantic Manipulation Figure 1: Multi-code GAN prior facilitates many image processing applications using the reconstruction from fixed PGGAN [23] models.

我们表明，诸如Stylegan和Biggan之类的预训练的生成对抗网络（GAN）可以用作潜在银行，以提高图像超分辨率的性能。尽管大多数现有面向感知的方法试图通过以对抗性损失学习来产生现实的产出，但我们的方法，即生成的潜在银行（GLEAN），通过直接利用预先训练的gan封装的丰富而多样的先验来超越现有实践。但是，与需要在运行时需要昂贵的图像特定优化的普遍的GAN反演方法不同，我们的方法只需要单个前向通行证才能修复。可以轻松地将Glean合并到具有多分辨率Skip连接的简单编码器银行decoder架构中。采用来自不同生成模型的先验，可以将收集到各种类别（例如人的面孔，猫，建筑物和汽车）。我们进一步提出了一个轻巧的Glean，名为Lightglean，该版本仅保留Glean中的关键组成部分。值得注意的是，Lightglean仅由21％的参数和35％的拖鞋组成，同时达到可比的图像质量。我们将方法扩展到不同的任务，包括图像着色和盲图恢复，广泛的实验表明，与现有方法相比，我们提出的模型表现出色。代码和模型可在https://github.com/open-mmlab/mmediting上找到。

Deep convolutional networks have become a popular tool for image generation and restoration. Generally, their excellent performance is imputed to their ability to learn realistic image priors from a large number of example images. In this paper, we show that, on the contrary, the structure of a generator network is sufficient to capture a great deal of low-level image statistics prior to any learning. In order to do so, we show that a randomly-initialized neural network can be used as a handcrafted prior with excellent results in standard inverse problems such as denoising, superresolution, and inpainting. Furthermore, the same prior can be used to invert deep neural representations to diagnose them, and to restore images based on flash-no flash input pairs.

The Super-Resolution Generative Adversarial Network (SR-GAN) [1] is a seminal work that is capable of generating realistic textures during single image super-resolution. However, the hallucinated details are often accompanied with unpleasant artifacts. To further enhance the visual quality, we thoroughly study three key components of SRGANnetwork architecture, adversarial loss and perceptual loss, and improve each of them to derive an Enhanced SRGAN (ESRGAN). In particular, we introduce the Residual-in-Residual Dense Block (RRDB) without batch normalization as the basic network building unit. Moreover, we borrow the idea from relativistic GAN [2] to let the discriminator predict relative realness instead of the absolute value. Finally, we improve the perceptual loss by using the features before activation, which could provide stronger supervision for brightness consistency and texture recovery. Benefiting from these improvements, the proposed ESRGAN achieves consistently better visual quality with more realistic and natural textures than SRGAN and won the first place in the PIRM2018-SR Challenge 1 [3]. The code is available at https://github.com/xinntao/ESRGAN.

从单个样本产生图像，作为图像合成的新发展分支，引起了广泛的关注。在本文中，我们将该问题与单个图像的条件分布进行采样，提出了一种分层框架，通过关于结构，语义和纹理的分布的连续学习来简化复杂条件分布的学习学习和一代可理解。在此基础上，我们设计由三个级联的GAN组成的Exsingan，用于从给定的图像学习可解释的生成模型，级联的GANS连续模拟结构，语义和纹理的分布。由于以前的作品所做的，但也是从给定图像的内部补丁来学习的，而且来自GaN反演技术的外部获得的外部。与先前作品相比，Exsingan对内部和外部信息的适当组合有利于内部和外部信息的适当组合，对图像操纵任务进行了更强大的生成和竞争泛化能力。

由于简单但有效的训练机制和出色的图像产生质量，生成的对抗网络（GAN）引起了极大的关注。具有生成照片现实的高分辨率（例如$ 1024 \ times1024 $）的能力，最近的GAN模型已大大缩小了生成的图像与真实图像之间的差距。因此，许多最近的作品表明，通过利用良好的潜在空间和博学的gan先验来利用预先训练的GAN模型的新兴兴趣。在本文中，我们简要回顾了从三个方面利用预先培训的大规模GAN模型的最新进展，即1）大规模生成对抗网络的培训，2）探索和理解预训练的GAN模型，以及预先培训的GAN模型，以及3）利用这些模型进行后续任务，例如图像恢复和编辑。有关相关方法和存储库的更多信息，请访问https://github.com/csmliu/pretretaining-gans。

Single image super-resolution is the task of inferring a high-resolution image from a single low-resolution input. Traditionally, the performance of algorithms for this task is measured using pixel-wise reconstruction measures such as peak signal-to-noise ratio (PSNR) which have been shown to correlate poorly with the human perception of image quality. As a result, algorithms minimizing these metrics tend to produce over-smoothed images that lack highfrequency textures and do not look natural despite yielding high PSNR values.We propose a novel application of automated texture synthesis in combination with a perceptual loss focusing on creating realistic textures rather than optimizing for a pixelaccurate reproduction of ground truth images during training. By using feed-forward fully convolutional neural networks in an adversarial training setting, we achieve a significant boost in image quality at high magnification ratios. Extensive experiments on a number of datasets show the effectiveness of our approach, yielding state-of-the-art results in both quantitative and qualitative benchmarks.

基于补丁的方法和深度网络已经采用了解决图像染色问题，具有自己的优势和劣势。基于补丁的方法能够通过从未遮盖区域搜索最近的邻居修补程序来恢复具有高质量纹理的缺失区域。但是，这些方法在恢复大缺失区域时会带来问题内容。另一方面，深度网络显示有希望的成果完成大区域。尽管如此，结果往往缺乏类似周围地区的忠诚和尖锐的细节。通过汇集两个范式中，我们提出了一种新的深度染色框架，其中纹理生成是由从未掩蔽区域提取的补丁样本的纹理记忆引导的。该框架具有一种新颖的设计，允许使用深度修复网络训练纹理存储器检索。此外，我们还介绍了贴片分配损失，以鼓励高质量的贴片合成。所提出的方法在三个具有挑战性的图像基准测试中，即地位，Celeba-HQ和巴黎街道视图数据集来说，该方法显示出质量和定量的卓越性能。

Face Restoration (FR) aims to restore High-Quality (HQ) faces from Low-Quality (LQ) input images, which is a domain-specific image restoration problem in the low-level computer vision area. The early face restoration methods mainly use statistic priors and degradation models, which are difficult to meet the requirements of real-world applications in practice. In recent years, face restoration has witnessed great progress after stepping into the deep learning era. However, there are few works to study deep learning-based face restoration methods systematically. Thus, this paper comprehensively surveys recent advances in deep learning techniques for face restoration. Specifically, we first summarize different problem formulations and analyze the characteristic of the face image. Second, we discuss the challenges of face restoration. Concerning these challenges, we present a comprehensive review of existing FR methods, including prior based methods and deep learning-based methods. Then, we explore developed techniques in the task of FR covering network architectures, loss functions, and benchmark datasets. We also conduct a systematic benchmark evaluation on representative methods. Finally, we discuss future directions, including network designs, metrics, benchmark datasets, applications,etc. We also provide an open-source repository for all the discussed methods, which is available at https://github.com/TaoWangzj/Awesome-Face-Restoration.

尽管无条件的特征反演是许多图像合成应用的基础，但训练逆变器需要高计算预算，大型解码容量和强加的条件，例如自回旋先验。为了解决这些局限性，我们建议使用对抗强大的表示作为特征反演的感知原始。我们训练一个对抗性稳健的编码器，以提取分离和感知对齐的图像表示，使其容易逆转。通过使用编码器的镜像架构训练简单的发电机，我们实现了优于标准模型的卓越重建质量和概括。基于此，我们提出了一个具有对抗性的自动编码器，并展示了其在样式转移，图像denoisising和异常检测任务方面的改进性能。与最近的Imagenet特征反演方法相比，我们的模型的性能提高了，复杂性的性能明显较小。

Our goal with this survey is to provide an overview of the state of the art deep learning technologies for face generation and editing. We will cover popular latest architectures and discuss key ideas that make them work, such as inversion, latent representation, loss functions, training procedures, editing methods, and cross domain style transfer. We particularly focus on GAN-based architectures that have culminated in the StyleGAN approaches, which allow generation of high-quality face images and offer rich interfaces for controllable semantics editing and preserving photo quality. We aim to provide an entry point into the field for readers that have basic knowledge about the field of deep learning and are looking for an accessible introduction and overview.

The primary aim of single-image super-resolution is to construct a high-resolution (HR) image from a corresponding low-resolution (LR) input. In previous approaches, which have generally been supervised, the training objective typically measures a pixel-wise average distance between the super-resolved (SR) and HR images. Optimizing such metrics often leads to blurring, especially in high variance (detailed) regions. We propose an alternative formulation of the super-resolution problem based on creating realistic SR images that downscale correctly. We present a novel super-resolution algorithm addressing this problem, PULSE (Photo Upsampling via Latent Space Exploration), which generates high-resolution, realistic images at resolutions previously unseen in the literature. It accomplishes this in an entirely self-supervised fashion and is not confined to a specific degradation operator used during training, unlike previous methods (which require training on databases of LR-HR image pairs for supervised learning). Instead of starting with the LR image and slowly adding detail, PULSE traverses the high-resolution natural image manifold, searching for images that downscale to the original LR image. This is formalized through the "downscaling loss," which guides exploration through the latent space of a generative model. By leveraging properties of high-dimensional Gaussians, we restrict the search space to guarantee that our outputs are realistic. PULSE thereby generates super-resolved images that both are realistic and downscale correctly. We show extensive experimental results demonstrating the efficacy of our approach in the domain of face super-resolution (also known as face hallucination). We also present a discussion of the limitations and biases of the method as currently implemented with an accompanying model card with relevant metrics. Our method outperforms state-of-the-art methods in perceptual quality at higher resolutions and scale factors than previously pos-sible.

The introduction of high-quality image generation models, particularly the StyleGAN family, provides a powerful tool to synthesize and manipulate images. However, existing models are built upon high-quality (HQ) data as desired outputs, making them unfit for in-the-wild low-quality (LQ) images, which are common inputs for manipulation. In this work, we bridge this gap by proposing a novel GAN structure that allows for generating images with controllable quality. The network can synthesize various image degradation and restore the sharp image via a quality control code. Our proposed QC-StyleGAN can directly edit LQ images without altering their quality by applying GAN inversion and manipulation techniques. It also provides for free an image restoration solution that can handle various degradations, including noise, blur, compression artifacts, and their mixtures. Finally, we demonstrate numerous other applications such as image degradation synthesis, transfer, and interpolation.

现实世界图像超分辨率（SR）的关键挑战是在低分辨率（LR）图像中恢复具有复杂未知降解（例如，下采样，噪声和压缩）的缺失细节。大多数以前的作品还原图像空间中的此类缺失细节。为了应对自然图像的高度多样性，他们要么依靠难以训练和容易训练和伪影的不稳定的甘体，要么诉诸于通常不可用的高分辨率（HR）图像中的明确参考。在这项工作中，我们提出了匹配SR（FEMASR）的功能，该功能在更紧凑的特征空间中恢复了现实的HR图像。与图像空间方法不同，我们的FEMASR通过将扭曲的LR图像{\ IT特征}与我们预读的HR先验中的无失真性HR对应物匹配来恢复HR图像，并解码匹配的功能以获得现实的HR图像。具体而言，我们的人力资源先验包含一个离散的特征代码簿及其相关的解码器，它们在使用量化的生成对抗网络（VQGAN）的HR图像上预估计。值得注意的是，我们在VQGAN中结合了一种新型的语义正则化，以提高重建图像的质量。对于功能匹配，我们首先提取由LR编码器组成的LR编码器的LR功能，然后遵循简单的最近邻居策略，将其与预读的代码簿匹配。特别是，我们为LR编码器配备了与解码器的残留快捷方式连接，这对于优化功能匹配损耗至关重要，还有助于补充可能的功能匹配错误。实验结果表明，我们的方法比以前的方法产生更现实的HR图像。代码以\ url {https://github.com/chaofengc/femasr}发布。

近年来，着色吸引了越来越多的兴趣。经典的基于参考的方法通常依靠外部颜色图像来获得合理的结果。检索此类示例不可避免地需要大型图像数据库或在线搜索引擎。最近的基于深度学习的方法可以自动以低成本为图像着色。但是，总是伴随着不满意的文物和不连贯的颜色。在这项工作中，我们提出了GCP颜色化，以利用预审前的生成对抗网络（GAN）封装的丰富和多样化的颜色先验进行自动着色。具体而言，我们首先通过GAN编码器“检索”匹配的功能（类似于示例），然后将这些功能与功能调制量合并到着色过程中。得益于强大的生成颜色先验（GCP）和精致的设计，我们的GCP颜色可以通过单个前向传球产生生动的颜色。此外，通过修改GAN潜在代码获得多样化的结果非常方便。 GCP颜色还继承了可解释的gan的功能，并可以通过穿过甘恩潜在空间来实现可控制和平滑的过渡。广泛的实验和用户研究表明，GCP颜色比以前的作品具有出色的性能。代码可在https://github.com/tothebeginning/gcp-colorization上找到。

Random samples from a single image Single training image Figure 1: Image generation learned from a single training image. We propose SinGAN-a new unconditional generative model trained on a single natural image. Our model learns the image's patch statistics across multiple scales, using a dedicated multi-scale adversarial training scheme; it can then be used to generate new realistic image samples that preserve the original patch distribution while creating new object configurations and structures.

在这项工作中，我们研究了面部重建的问题，鉴于从黑框面部识别引擎中提取的面部特征表示。确实，由于引擎中抽象信息的局限性，在实践中，这是非常具有挑战性的问题。因此，我们在蒸馏框架（dab-gan）中引入了一种名为基于注意力的生成对抗网络的新方法，以合成受试者的面孔，鉴于其提取的面部识别功能。鉴于主题的任何不受约束的面部特征，Dab-Gan可以在高清上重建他/她的脸。 DAB-GAN方法包括一种新型的基于注意力的生成结构，采用新的定义的Bioxtive Metrics学习方法。该框架首先引入徒图，以便可以在图像域中直接采用距离测量和度量学习过程，以进行图像重建任务。来自Blackbox面部识别引擎的信息将使用全局蒸馏过程最佳利用。然后，提出了一个基于注意力的发电机，以使一个高度可靠的发电机通过ID保存综合逼真的面孔。我们已经评估了有关具有挑战性的面部识别数据库的方法，即Celeba，LF​​W，AgeDB，CFP-FP，并始终取得了最新的结果。 Dab-Gan的进步也得到了图像现实主义和ID保存属性的证明。

Despite the breakthroughs in accuracy and speed of single image super-resolution using faster and deeper convolutional neural networks, one central problem remains largely unsolved: how do we recover the finer texture details when we super-resolve at large upscaling factors? The behavior of optimization-based super-resolution methods is principally driven by the choice of the objective function. Recent work has largely focused on minimizing the mean squared reconstruction error. The resulting estimates have high peak signal-to-noise ratios, but they are often lacking high-frequency details and are perceptually unsatisfying in the sense that they fail to match the fidelity expected at the higher resolution. In this paper, we present SRGAN, a generative adversarial network (GAN) for image superresolution (SR). To our knowledge, it is the first framework capable of inferring photo-realistic natural images for 4× upscaling factors. To achieve this, we propose a perceptual loss function which consists of an adversarial loss and a content loss. The adversarial loss pushes our solution to the natural image manifold using a discriminator network that is trained to differentiate between the super-resolved images and original photo-realistic images. In addition, we use a content loss motivated by perceptual similarity instead of similarity in pixel space. Our deep residual network is able to recover photo-realistic textures from heavily downsampled images on public benchmarks. An extensive mean-opinion-score (MOS) test shows hugely significant gains in perceptual quality using SRGAN. The MOS scores obtained with SRGAN are closer to those of the original high-resolution images than to those obtained with any state-of-the-art method.

Deep learning techniques have made considerable progress in image inpainting, restoration, and reconstruction in the last few years. Image outpainting, also known as image extrapolation, lacks attention and practical approaches to be fulfilled, owing to difficulties caused by large-scale area loss and less legitimate neighboring information. These difficulties have made outpainted images handled by most of the existing models unrealistic to human eyes and spatially inconsistent. When upsampling through deconvolution to generate fake content, the naive generation methods may lead to results lacking high-frequency details and structural authenticity. Therefore, as our novelties to handle image outpainting problems, we introduce structural prior as a condition to optimize the generation quality and a new semantic embedding term to enhance perceptual sanity. we propose a deep learning method based on Generative Adversarial Network (GAN) and condition edges as structural prior in order to assist the generation. We use a multi-phase adversarial training scheme that comprises edge inference training, contents inpainting training, and joint training. The newly added semantic embedding loss is proved effective in practice.