edu.hk (a) Image Reconstruction (b) Image Colorization (c) Image Super-Resolution (d) Image Denoising (e) Image Inpainting (f) Semantic Manipulation Figure 1: Multi-code GAN prior facilitates many image processing applications using the reconstruction from fixed PGGAN [23] models.

Recent work has shown that a variety of semantics emerge in the latent space of Generative Adversarial Networks (GANs) when being trained to synthesize images. However, it is difficult to use these learned semantics for real image editing. A common practice of feeding a real image to a trained GAN generator is to invert it back to a latent code. However, existing inversion methods typically focus on reconstructing the target image by pixel values yet fail to land the inverted code in the semantic domain of the original latent space. As a result, the reconstructed image cannot well support semantic editing through varying the inverted code. To solve this problem, we propose an in-domain GAN inversion approach, which not only faithfully reconstructs the input image but also ensures the inverted code to be semantically meaningful for editing. We first learn a novel domain-guided encoder to project a given image to the native latent space of GANs. We then propose domain-regularized optimization by involving the encoder as a regularizer to fine-tune the code produced by the encoder and better recover the target image. Extensive experiments suggest that our inversion method achieves satisfying real image reconstruction and more importantly facilitates various image editing tasks, significantly outperforming start-of-the-arts. 1

Learning a good image prior is a long-term goal for image restoration and manipulation. While existing methods like deep image prior (DIP) capture low-level image statistics, there are still gaps toward an image prior that captures rich image semantics including color, spatial coherence, textures, and high-level concepts. This work presents an effective way to exploit the image prior captured by a generative adversarial network (GAN) trained on large-scale natural images. As shown in Fig. 1, the deep generative prior (DGP) provides compelling results to restore missing semantics, e.g., color, patch, resolution, of various degraded images. It also enables diverse image manipulation including random jittering, image morphing, and category transfer. Such highly flexible restoration and manipulation are made possible through relaxing the assumption of existing GAN-inversion methods, which tend to fix the generator. Notably, we allow the generator to be fine-tuned on-the-fly in a progressive manner regularized by feature distance obtained by the discriminator in GAN. We show that these easy-to-implement and practical changes help preserve the reconstruction to remain in the manifold of nature image, and thus lead to more precise and faithful reconstruction for real images. Code is available at https://github.com/XingangPan/deepgenerative-prior.

图像介入寻求一种语义一致的方法，以根据其未掩盖的内容来恢复损坏的图像。以前的方法通常将训练有素的甘恩重复使用，然后在产生逼真的斑块中用于缺少GAN反转的孔。然而，在这些算法中对硬约束的无知可能会产生gan倒置和图像插入之间的差距。在解决这个问题的情况下，我们在本文中设计了一个新颖的GAN反转模型，用于图像插入，称为Interverfill，主要由带有预调制模块的编码器和具有F＆W+潜在空间的GAN生成器组成。在编码器中，预调制网络利用多尺度结构将更多的歧视语义编码为样式向量。为了弥合GAN倒置和图像插入之间的缝隙，提出了F＆W+潜在空间以消除巨大的颜色差异和语义不一致。为了重建忠实和逼真的图像，一个简单而有效的软上升平均潜在模块旨在捕获更多样化的内域模式，以合成大型腐败的高保真质地。在包括Ploce2，Celeba-HQ，Metfaces和Scenery在内的四个具有挑战性的数据集上进行的全面实验表明，我们的Intervill效果优于定性和定量的高级方法，并支持室外图像的完成。

Figure 1. The proposed pixel2style2pixel framework can be used to solve a wide variety of image-to-image translation tasks. Here we show results of pSp on StyleGAN inversion, multi-modal conditional image synthesis, facial frontalization, inpainting and super-resolution.

反转生成对抗网络（GAN）可以使用预训练的发电机来促进广泛的图像编辑任务。现有方法通常采用gan的潜在空间作为反转空间，但观察到空间细节的恢复不足。在这项工作中，我们建议涉及发电机的填充空间，以通过空间信息补充潜在空间。具体来说，我们替换具有某些实例感知系数的卷积层中使用的恒定填充（例如，通常为零）。通过这种方式，可以适当地适当地适应了预训练模型中假定的归纳偏差以适合每个单独的图像。通过学习精心设计的编码器，我们设法在定性和定量上提高了反演质量，超过了现有的替代方案。然后，我们证明了这样的空间扩展几乎不会影响天然甘纳的歧管，因此我们仍然可以重复使用甘斯（Gans）对各种下游应用学到的先验知识。除了在先前的艺术中探讨的编辑任务外，我们的方法还可以进行更灵活的图像操纵，例如对面部轮廓和面部细节的单独控制，并启用一种新颖的编辑方式，用户可以高效地自定义自己的操作。

由于简单但有效的训练机制和出色的图像产生质量，生成的对抗网络（GAN）引起了极大的关注。具有生成照片现实的高分辨率（例如$ 1024 \ times1024 $）的能力，最近的GAN模型已大大缩小了生成的图像与真实图像之间的差距。因此，许多最近的作品表明，通过利用良好的潜在空间和博学的gan先验来利用预先训练的GAN模型的新兴兴趣。在本文中，我们简要回顾了从三个方面利用预先培训的大规模GAN模型的最新进展，即1）大规模生成对抗网络的培训，2）探索和理解预训练的GAN模型，以及预先培训的GAN模型，以及3）利用这些模型进行后续任务，例如图像恢复和编辑。有关相关方法和存储库的更多信息，请访问https://github.com/csmliu/pretretaining-gans。

The introduction of high-quality image generation models, particularly the StyleGAN family, provides a powerful tool to synthesize and manipulate images. However, existing models are built upon high-quality (HQ) data as desired outputs, making them unfit for in-the-wild low-quality (LQ) images, which are common inputs for manipulation. In this work, we bridge this gap by proposing a novel GAN structure that allows for generating images with controllable quality. The network can synthesize various image degradation and restore the sharp image via a quality control code. Our proposed QC-StyleGAN can directly edit LQ images without altering their quality by applying GAN inversion and manipulation techniques. It also provides for free an image restoration solution that can handle various degradations, including noise, blur, compression artifacts, and their mixtures. Finally, we demonstrate numerous other applications such as image degradation synthesis, transfer, and interpolation.

Face Restoration (FR) aims to restore High-Quality (HQ) faces from Low-Quality (LQ) input images, which is a domain-specific image restoration problem in the low-level computer vision area. The early face restoration methods mainly use statistic priors and degradation models, which are difficult to meet the requirements of real-world applications in practice. In recent years, face restoration has witnessed great progress after stepping into the deep learning era. However, there are few works to study deep learning-based face restoration methods systematically. Thus, this paper comprehensively surveys recent advances in deep learning techniques for face restoration. Specifically, we first summarize different problem formulations and analyze the characteristic of the face image. Second, we discuss the challenges of face restoration. Concerning these challenges, we present a comprehensive review of existing FR methods, including prior based methods and deep learning-based methods. Then, we explore developed techniques in the task of FR covering network architectures, loss functions, and benchmark datasets. We also conduct a systematic benchmark evaluation on representative methods. Finally, we discuss future directions, including network designs, metrics, benchmark datasets, applications,etc. We also provide an open-source repository for all the discussed methods, which is available at https://github.com/TaoWangzj/Awesome-Face-Restoration.

Although Generative Adversarial Networks (GANs) have made significant progress in face synthesis, there lacks enough understanding of what GANs have learned in the latent representation to map a random code to a photo-realistic image. In this work, we propose a framework called InterFaceGAN to interpret the disentangled face representation learned by the state-of-the-art GAN models and study the properties of the facial semantics encoded in the latent space. We first find that GANs learn various semantics in some linear subspaces of the latent space. After identifying these subspaces, we can realistically manipulate the corresponding facial attributes without retraining the model. We then conduct a detailed study on the correlation between different semantics and manage to better disentangle them via subspace projection, resulting in more precise control of the attribute manipulation. Besides manipulating the gender, age, expression, and presence of eyeglasses, we can even alter the face pose and fix the artifacts accidentally made by GANs. Furthermore, we perform an in-depth face identity analysis and a layer-wise analysis to evaluate the editing results quantitatively. Finally, we apply our approach to real face editing by employing GAN inversion approaches and explicitly training feed-forward models based on the synthetic data established by InterFaceGAN. Extensive experimental results suggest that learning to synthesize faces spontaneously brings a disentangled and controllable face representation.

We present a novel image inversion framework and a training pipeline to achieve high-fidelity image inversion with high-quality attribute editing. Inverting real images into StyleGAN's latent space is an extensively studied problem, yet the trade-off between the image reconstruction fidelity and image editing quality remains an open challenge. The low-rate latent spaces are limited in their expressiveness power for high-fidelity reconstruction. On the other hand, high-rate latent spaces result in degradation in editing quality. In this work, to achieve high-fidelity inversion, we learn residual features in higher latent codes that lower latent codes were not able to encode. This enables preserving image details in reconstruction. To achieve high-quality editing, we learn how to transform the residual features for adapting to manipulations in latent codes. We train the framework to extract residual features and transform them via a novel architecture pipeline and cycle consistency losses. We run extensive experiments and compare our method with state-of-the-art inversion methods. Qualitative metrics and visual comparisons show significant improvements. Code: https://github.com/hamzapehlivan/StyleRes

现有的GAN倒置和编辑方法适用于具有干净背景的对齐物体，例如肖像和动物面孔，但通常会为更加困难的类别而苦苦挣扎，具有复杂的场景布局和物体遮挡，例如汽车，动物和室外图像。我们提出了一种新方法，以在gan的潜在空间（例如stylegan2）中倒转和编辑复杂的图像。我们的关键想法是用一系列层的集合探索反演，从而将反转过程适应图像的难度。我们学会预测不同图像段的“可逆性”，并将每个段投影到潜在层。更容易的区域可以倒入发电机潜在空间中的较早层，而更具挑战性的区域可以倒入更晚的特征空间。实验表明，与最新的复杂类别的方法相比，我们的方法获得了更好的反转结果，同时保持下游的编辑性。请参阅我们的项目页面，网址为https://www.cs.cmu.edu/~saminversion。

我们表明，诸如Stylegan和Biggan之类的预训练的生成对抗网络（GAN）可以用作潜在银行，以提高图像超分辨率的性能。尽管大多数现有面向感知的方法试图通过以对抗性损失学习来产生现实的产出，但我们的方法，即生成的潜在银行（GLEAN），通过直接利用预先训练的gan封装的丰富而多样的先验来超越现有实践。但是，与需要在运行时需要昂贵的图像特定优化的普遍的GAN反演方法不同，我们的方法只需要单个前向通行证才能修复。可以轻松地将Glean合并到具有多分辨率Skip连接的简单编码器银行decoder架构中。采用来自不同生成模型的先验，可以将收集到各种类别（例如人的面孔，猫，建筑物和汽车）。我们进一步提出了一个轻巧的Glean，名为Lightglean，该版本仅保留Glean中的关键组成部分。值得注意的是，Lightglean仅由21％的参数和35％的拖鞋组成，同时达到可比的图像质量。我们将方法扩展到不同的任务，包括图像着色和盲图恢复，广泛的实验表明，与现有方法相比，我们提出的模型表现出色。代码和模型可在https://github.com/open-mmlab/mmediting上找到。

In this work, we propose TediGAN, a novel framework for multi-modal image generation and manipulation with textual descriptions. The proposed method consists of three components: StyleGAN inversion module, visual-linguistic similarity learning, and instance-level optimization. The inversion module maps real images to the latent space of a well-trained StyleGAN. The visual-linguistic similarity learns the text-image matching by mapping the image and text into a common embedding space. The instancelevel optimization is for identity preservation in manipulation. Our model can produce diverse and high-quality images with an unprecedented resolution at 1024 2 . Using a control mechanism based on style-mixing, our Tedi-GAN inherently supports image synthesis with multi-modal inputs, such as sketches or semantic labels, with or without instance guidance. To facilitate text-guided multimodal synthesis, we propose the Multi-Modal CelebA-HQ, a large-scale dataset consisting of real face images and corresponding semantic segmentation map, sketch, and textual descriptions. Extensive experiments on the introduced dataset demonstrate the superior performance of our proposed method. Code and data are available at https://github.com/weihaox/TediGAN.

A rich set of interpretable dimensions has been shown to emerge in the latent space of the Generative Adversarial Networks (GANs) trained for synthesizing images. In order to identify such latent dimensions for image editing, previous methods typically annotate a collection of synthesized samples and train linear classifiers in the latent space. However, they require a clear definition of the target attribute as well as the corresponding manual annotations, limiting their applications in practice. In this work, we examine the internal representation learned by GANs to reveal the underlying variation factors in an unsupervised manner. In particular, we take a closer look into the generation mechanism of GANs and further propose a closed-form factorization algorithm for latent semantic discovery by directly decomposing the pre-trained weights. With a lightning-fast implementation, our approach is capable of not only finding semantically meaningful dimensions comparably to the state-of-the-art supervised methods, but also resulting in far more versatile concepts across multiple GAN models trained on a wide range of datasets. 1

生成照片 - 现实图像，语义编辑和表示学习是高分辨率生成模型的许多潜在应用中的一些。最近在GAN的进展将它们建立为这些任务的绝佳选择。但是，由于它们不提供推理模型，因此使用GaN潜在空间无法在实际图像上完成诸如分类的图像编辑或下游任务。尽管培训了训练推理模型或设计了一种迭代方法来颠覆训练有素的发生器，但之前的方法是数据集（例如人类脸部图像）和架构（例如样式）。这些方法是非延伸到新型数据集或架构的。我们提出了一般框架，该框架是不可知的架构和数据集。我们的主要识别是，通过培训推断和生成模型在一起，我们允许它们彼此适应并收敛到更好的质量模型。我们的\ textbf {invang}，可逆GaN的简短，成功将真实图像嵌入到高质量的生成模型的潜在空间。这使我们能够执行图像修复，合并，插值和在线数据增强。我们展示了广泛的定性和定量实验。

通过利用预熟gan的潜在空间，已经提出了许多最近的作品来进行面部图像编辑。但是，很少有尝试将它们直接应用于视频，因为1）他们不能保证时间一致性，2）他们的应用受到视频的处理速度的限制，3）他们无法准确编码面部运动和表达的细节。为此，我们提出了一个新颖的网络，将面部视频编码到Stylegan的潜在空间中，以进行语义面部视频操纵。基于视觉变压器，我们的网络重复了潜在向量的高分辨率部分，以实现时间一致性。为了捕捉微妙的面部运动和表情，我们设计了涉及稀疏面部地标和密集的3D脸部网眼的新颖损失。我们已经彻底评估了我们的方法，并成功证明了其对各种面部视频操作的应用。特别是，我们提出了一个新型网络，用于3D坐标系中的姿势/表达控制。定性和定量结果都表明，我们的方法可以显着优于现有的单图方法，同时实现实时（66 fps）速度。

现实世界图像超分辨率（SR）的关键挑战是在低分辨率（LR）图像中恢复具有复杂未知降解（例如，下采样，噪声和压缩）的缺失细节。大多数以前的作品还原图像空间中的此类缺失细节。为了应对自然图像的高度多样性，他们要么依靠难以训练和容易训练和伪影的不稳定的甘体，要么诉诸于通常不可用的高分辨率（HR）图像中的明确参考。在这项工作中，我们提出了匹配SR（FEMASR）的功能，该功能在更紧凑的特征空间中恢复了现实的HR图像。与图像空间方法不同，我们的FEMASR通过将扭曲的LR图像{\ IT特征}与我们预读的HR先验中的无失真性HR对应物匹配来恢复HR图像，并解码匹配的功能以获得现实的HR图像。具体而言，我们的人力资源先验包含一个离散的特征代码簿及其相关的解码器，它们在使用量化的生成对抗网络（VQGAN）的HR图像上预估计。值得注意的是，我们在VQGAN中结合了一种新型的语义正则化，以提高重建图像的质量。对于功能匹配，我们首先提取由LR编码器组成的LR编码器的LR功能，然后遵循简单的最近邻居策略，将其与预读的代码簿匹配。特别是，我们为LR编码器配备了与解码器的残留快捷方式连接，这对于优化功能匹配损耗至关重要，还有助于补充可能的功能匹配错误。实验结果表明，我们的方法比以前的方法产生更现实的HR图像。代码以\ url {https://github.com/chaofengc/femasr}发布。

Our goal with this survey is to provide an overview of the state of the art deep learning technologies for face generation and editing. We will cover popular latest architectures and discuss key ideas that make them work, such as inversion, latent representation, loss functions, training procedures, editing methods, and cross domain style transfer. We particularly focus on GAN-based architectures that have culminated in the StyleGAN approaches, which allow generation of high-quality face images and offer rich interfaces for controllable semantics editing and preserving photo quality. We aim to provide an entry point into the field for readers that have basic knowledge about the field of deep learning and are looking for an accessible introduction and overview.

我们提出了Exe-Gan，这是一种新型的使用生成对抗网络的典范引导的面部介绍框架。我们的方法不仅可以保留输入面部图像的质量，而且还可以使用类似示例性的面部属性来完成图像。我们通过同时利用输入图像的全局样式，从随机潜在代码生成的随机样式以及示例图像的示例样式来实现这一目标。我们介绍了一个新颖的属性相似性指标，以鼓励网络以一种自我监督的方式从示例中学习面部属性的风格。为了确保跨地区边界之间的自然过渡，我们引入了一种新型的空间变体梯度反向传播技术，以根据空间位置调整损耗梯度。关于公共Celeba-HQ和FFHQ数据集的广泛评估和实际应用，可以验证Exe-GAN的优越性，从面部镶嵌的视觉质量来看。