生成照片 - 现实图像，语义编辑和表示学习是高分辨率生成模型的许多潜在应用中的一些。最近在GAN的进展将它们建立为这些任务的绝佳选择。但是，由于它们不提供推理模型，因此使用GaN潜在空间无法在实际图像上完成诸如分类的图像编辑或下游任务。尽管培训了训练推理模型或设计了一种迭代方法来颠覆训练有素的发生器，但之前的方法是数据集（例如人类脸部图像）和架构（例如样式）。这些方法是非延伸到新型数据集或架构的。我们提出了一般框架，该框架是不可知的架构和数据集。我们的主要识别是，通过培训推断和生成模型在一起，我们允许它们彼此适应并收敛到更好的质量模型。我们的\ textbf {invang}，可逆GaN的简短，成功将真实图像嵌入到高质量的生成模型的潜在空间。这使我们能够执行图像修复，合并，插值和在线数据增强。我们展示了广泛的定性和定量实验。

Our goal with this survey is to provide an overview of the state of the art deep learning technologies for face generation and editing. We will cover popular latest architectures and discuss key ideas that make them work, such as inversion, latent representation, loss functions, training procedures, editing methods, and cross domain style transfer. We particularly focus on GAN-based architectures that have culminated in the StyleGAN approaches, which allow generation of high-quality face images and offer rich interfaces for controllable semantics editing and preserving photo quality. We aim to provide an entry point into the field for readers that have basic knowledge about the field of deep learning and are looking for an accessible introduction and overview.

扩散概率模型（DPMS）在竞争对手GANS的图像生成中取得了显着的质量。但与GAN不同，DPMS使用一组缺乏语义含义的一组潜在变量，并且不能作为其他任务的有用表示。本文探讨了使用DPMS进行表示学习的可能性，并寻求通过自动编码提取输入图像的有意义和可解码的表示。我们的主要思想是使用可学习的编码器来发现高级语义，以及DPM作为用于建模剩余随机变化的解码器。我们的方法可以将任何图像编码为两部分潜在的代码，其中第一部分是语义有意义和线性的，第二部分捕获随机细节，允许接近精确的重建。这种功能使当前箔基于GaN的方法的挑战性应用，例如实际图像上的属性操作。我们还表明，这两级编码可提高去噪效率，自然地涉及各种下游任务，包括几次射击条件采样。

由于简单但有效的训练机制和出色的图像产生质量，生成的对抗网络（GAN）引起了极大的关注。具有生成照片现实的高分辨率（例如$ 1024 \ times1024 $）的能力，最近的GAN模型已大大缩小了生成的图像与真实图像之间的差距。因此，许多最近的作品表明，通过利用良好的潜在空间和博学的gan先验来利用预先训练的GAN模型的新兴兴趣。在本文中，我们简要回顾了从三个方面利用预先培训的大规模GAN模型的最新进展，即1）大规模生成对抗网络的培训，2）探索和理解预训练的GAN模型，以及预先培训的GAN模型，以及3）利用这些模型进行后续任务，例如图像恢复和编辑。有关相关方法和存储库的更多信息，请访问https://github.com/csmliu/pretretaining-gans。

We present a novel image inversion framework and a training pipeline to achieve high-fidelity image inversion with high-quality attribute editing. Inverting real images into StyleGAN's latent space is an extensively studied problem, yet the trade-off between the image reconstruction fidelity and image editing quality remains an open challenge. The low-rate latent spaces are limited in their expressiveness power for high-fidelity reconstruction. On the other hand, high-rate latent spaces result in degradation in editing quality. In this work, to achieve high-fidelity inversion, we learn residual features in higher latent codes that lower latent codes were not able to encode. This enables preserving image details in reconstruction. To achieve high-quality editing, we learn how to transform the residual features for adapting to manipulations in latent codes. We train the framework to extract residual features and transform them via a novel architecture pipeline and cycle consistency losses. We run extensive experiments and compare our method with state-of-the-art inversion methods. Qualitative metrics and visual comparisons show significant improvements. Code: https://github.com/hamzapehlivan/StyleRes

Figure 1. The proposed pixel2style2pixel framework can be used to solve a wide variety of image-to-image translation tasks. Here we show results of pSp on StyleGAN inversion, multi-modal conditional image synthesis, facial frontalization, inpainting and super-resolution.

Recent work has shown that a variety of semantics emerge in the latent space of Generative Adversarial Networks (GANs) when being trained to synthesize images. However, it is difficult to use these learned semantics for real image editing. A common practice of feeding a real image to a trained GAN generator is to invert it back to a latent code. However, existing inversion methods typically focus on reconstructing the target image by pixel values yet fail to land the inverted code in the semantic domain of the original latent space. As a result, the reconstructed image cannot well support semantic editing through varying the inverted code. To solve this problem, we propose an in-domain GAN inversion approach, which not only faithfully reconstructs the input image but also ensures the inverted code to be semantically meaningful for editing. We first learn a novel domain-guided encoder to project a given image to the native latent space of GANs. We then propose domain-regularized optimization by involving the encoder as a regularizer to fine-tune the code produced by the encoder and better recover the target image. Extensive experiments suggest that our inversion method achieves satisfying real image reconstruction and more importantly facilitates various image editing tasks, significantly outperforming start-of-the-arts. 1

edu.hk (a) Image Reconstruction (b) Image Colorization (c) Image Super-Resolution (d) Image Denoising (e) Image Inpainting (f) Semantic Manipulation Figure 1: Multi-code GAN prior facilitates many image processing applications using the reconstruction from fixed PGGAN [23] models.

The introduction of high-quality image generation models, particularly the StyleGAN family, provides a powerful tool to synthesize and manipulate images. However, existing models are built upon high-quality (HQ) data as desired outputs, making them unfit for in-the-wild low-quality (LQ) images, which are common inputs for manipulation. In this work, we bridge this gap by proposing a novel GAN structure that allows for generating images with controllable quality. The network can synthesize various image degradation and restore the sharp image via a quality control code. Our proposed QC-StyleGAN can directly edit LQ images without altering their quality by applying GAN inversion and manipulation techniques. It also provides for free an image restoration solution that can handle various degradations, including noise, blur, compression artifacts, and their mixtures. Finally, we demonstrate numerous other applications such as image degradation synthesis, transfer, and interpolation.

潜在矢量生成模型的潜在空间中数据点的不同编码可能会导致数据背后的不同解释因素的效率或多或少有效且分开的特征。最近，许多作品都致力于探索特定模型的潜在空间，主要集中在研究特征如何分离以及如何在可见空间中产生所需数据变化的轨迹变化。在这项工作中，我们解决了比较不同模型的潜在空间的更一般问题，寻找它们之间的转换。我们将调查局限于人脸数据歧管的熟悉且在很大程度上研究的生成模型案例。本文报道的令人惊讶的初步结果是（前提是（前提是模型尚未被教导或明确地想象以不同的方式采取行动）简单的线性映射足以从潜在空间传递到另一个信息，同时保留大多数信息。

Generative adversarial networks (GANs) provide a way to learn deep representations without extensively annotated training data. They achieve this through deriving backpropagation signals through a competitive process involving a pair of networks. The representations that can be learned by GANs may be used in a variety of applications, including image synthesis, semantic image editing, style transfer, image super-resolution and classification. The aim of this review paper is to provide an overview of GANs for the signal processing community, drawing on familiar analogies and concepts where possible. In addition to identifying different methods for training and constructing GANs, we also point to remaining challenges in their theory and application.

The primary aim of single-image super-resolution is to construct a high-resolution (HR) image from a corresponding low-resolution (LR) input. In previous approaches, which have generally been supervised, the training objective typically measures a pixel-wise average distance between the super-resolved (SR) and HR images. Optimizing such metrics often leads to blurring, especially in high variance (detailed) regions. We propose an alternative formulation of the super-resolution problem based on creating realistic SR images that downscale correctly. We present a novel super-resolution algorithm addressing this problem, PULSE (Photo Upsampling via Latent Space Exploration), which generates high-resolution, realistic images at resolutions previously unseen in the literature. It accomplishes this in an entirely self-supervised fashion and is not confined to a specific degradation operator used during training, unlike previous methods (which require training on databases of LR-HR image pairs for supervised learning). Instead of starting with the LR image and slowly adding detail, PULSE traverses the high-resolution natural image manifold, searching for images that downscale to the original LR image. This is formalized through the "downscaling loss," which guides exploration through the latent space of a generative model. By leveraging properties of high-dimensional Gaussians, we restrict the search space to guarantee that our outputs are realistic. PULSE thereby generates super-resolved images that both are realistic and downscale correctly. We show extensive experimental results demonstrating the efficacy of our approach in the domain of face super-resolution (also known as face hallucination). We also present a discussion of the limitations and biases of the method as currently implemented with an accompanying model card with relevant metrics. Our method outperforms state-of-the-art methods in perceptual quality at higher resolutions and scale factors than previously pos-sible.

Recent 3D-aware GANs rely on volumetric rendering techniques to disentangle the pose and appearance of objects, de facto generating entire 3D volumes rather than single-view 2D images from a latent code. Complex image editing tasks can be performed in standard 2D-based GANs (e.g., StyleGAN models) as manipulation of latent dimensions. However, to the best of our knowledge, similar properties have only been partially explored for 3D-aware GAN models. This work aims to fill this gap by showing the limitations of existing methods and proposing LatentSwap3D, a model-agnostic approach designed to enable attribute editing in the latent space of pre-trained 3D-aware GANs. We first identify the most relevant dimensions in the latent space of the model controlling the targeted attribute by relying on the feature importance ranking of a random forest classifier. Then, to apply the transformation, we swap the top-K most relevant latent dimensions of the image being edited with an image exhibiting the desired attribute. Despite its simplicity, LatentSwap3D provides remarkable semantic edits in a disentangled manner and outperforms alternative approaches both qualitatively and quantitatively. We demonstrate our semantic edit approach on various 3D-aware generative models such as pi-GAN, GIRAFFE, StyleSDF, MVCGAN, EG3D and VolumeGAN, and on diverse datasets, such as FFHQ, AFHQ, Cats, MetFaces, and CompCars. The project page can be found: \url{https://enisimsar.github.io/latentswap3d/}.

由于GaN潜在空间的勘探和利用，近年来，现实世界的图像操纵实现了奇妙的进展。 GaN反演是该管道的第一步，旨在忠实地将真实图像映射到潜在代码。不幸的是，大多数现有的GaN反演方法都无法满足下面列出的三个要求中的至少一个：重建质量，可编辑性和快速推断。我们在本研究中提出了一种新的两阶段策略，同时适合所有要求。在第一阶段，我们训练编码器将输入图像映射到StyleGan2 $ \ Mathcal {W} $ - 空间，这被证明具有出色的可编辑性，但重建质量较低。在第二阶段，我们通过利用一系列HyperNetWorks来补充初始阶段的重建能力以在反转期间恢复缺失的信息。这两个步骤互相补充，由于Hypernetwork分支和由于$ \ Mathcal {W} $ - 空间中的反转，因此由于HyperNetwork分支和优异的可编辑性而相互作用。我们的方法完全是基于编码器的，导致极快的推断。关于两个具有挑战性的数据集的广泛实验证明了我们方法的优越性。

反转生成对抗网络（GAN）可以使用预训练的发电机来促进广泛的图像编辑任务。现有方法通常采用gan的潜在空间作为反转空间，但观察到空间细节的恢复不足。在这项工作中，我们建议涉及发电机的填充空间，以通过空间信息补充潜在空间。具体来说，我们替换具有某些实例感知系数的卷积层中使用的恒定填充（例如，通常为零）。通过这种方式，可以适当地适当地适应了预训练模型中假定的归纳偏差以适合每个单独的图像。通过学习精心设计的编码器，我们设法在定性和定量上提高了反演质量，超过了现有的替代方案。然后，我们证明了这样的空间扩展几乎不会影响天然甘纳的歧管，因此我们仍然可以重复使用甘斯（Gans）对各种下游应用学到的先验知识。除了在先前的艺术中探讨的编辑任务外，我们的方法还可以进行更灵活的图像操纵，例如对面部轮廓和面部细节的单独控制，并启用一种新颖的编辑方式，用户可以高效地自定义自己的操作。

GAN的进展使高分辨率的感性质量形象产生了产生。 stylegans允许通过数学操作对W/W+空间中的潜在样式向量进行数学操作进行引人入胜的属性修改，从而有效调节生成器的丰富层次结构表示。最近，此类操作已被推广到原始StyleGan纸中的属性交换之外，以包括插值。尽管StyleGans有许多重大改进，但仍被认为会产生不自然的图像。生成的图像的质量基于两个假设。 （a）生成器学到的层次表示的丰富性，以及（b）样式空间的线性和平滑度。在这项工作中，我们提出了一个层次的语义正常化程序（HSR），该层次正常化程序将生成器学到的层次表示与大量数据学到的相应的强大功能保持一致。 HSR不仅可以改善发电机的表示，还可以改善潜在风格空间的线性和平滑度，从而导致产生更自然的样式编辑的图像。为了证明线性改善，我们提出了一种新型的度量 - 属性线性评分（ALS）。通过改善感知路径长度（PPL）度量的改善，在不同的标准数据集中平均16.19％的不自然图像的生成显着降低，同时改善了属性编辑任务中属性变化的线性变化。

随着脑成像技术和机器学习工具的出现，很多努力都致力于构建计算模型来捕获人脑中的视觉信息的编码。最具挑战性的大脑解码任务之一是通过功能磁共振成像（FMRI）测量的脑活动的感知自然图像的精确重建。在这项工作中，我们调查了来自FMRI的自然图像重建的最新学习方法。我们在架构设计，基准数据集和评估指标方面检查这些方法，并在标准化评估指标上呈现公平的性能评估。最后，我们讨论了现有研究的优势和局限，并提出了潜在的未来方向。

生成对抗网络（GAN）具有许多潜在的医学成像应用，包括数据扩展，域适应和模型解释。由于图形处理单元（GPU）的记忆力有限，因此在低分辨率的医学图像上对当前的3D GAN模型进行了训练，因此这些模型要么无法扩展到高分辨率，要么容易出现斑驳的人工制品。在这项工作中，我们提出了一种新颖的端到端GAN体系结构，可以生成高分辨率3D图像。我们通过使用训练和推理之间的不同配置来实现这一目标。在训练过程中，我们采用了层次结构，该结构同时生成图像的低分辨率版本和高分辨率图像的随机选择子量。层次设计具有两个优点：首先，对高分辨率图像训练的记忆需求在子量之间摊销。此外，将高分辨率子体积固定在单个低分辨率图像上可确保子量化之间的解剖一致性。在推断期间，我们的模型可以直接生成完整的高分辨率图像。我们还将具有类似层次结构的编码器纳入模型中，以从图像中提取特征。 3D胸CT和脑MRI的实验表明，我们的方法在图像生成中的表现优于最新技术。我们还证明了所提出的模型在数据增强和临床相关特征提取中的临床应用。

尽管使用StyleGan进行语义操纵的最新进展，但对真实面孔的语义编辑仍然具有挑战性。 $ W $空间与$ W $+空间之间的差距需要重建质量与编辑质量之间的不良权衡。为了解决这个问题，我们建议通过用基于注意的变压器代替Stylegan映射网络中的完全连接的层来扩展潜在空间。这种简单有效的技术将上述两个空间整合在一起，并将它们转换为一个名为$ W $ ++的新的潜在空间。我们的修改后的Stylegan保持了原始StyleGan的最新一代质量，并具有中等程度的多样性。但更重要的是，提议的$ W $ ++空间在重建质量和编辑质量方面都取得了卓越的性能。尽管有这些显着优势，但我们的$ W $ ++空间支持现有的反转算法和编辑方法，仅由于其与$ w/w $+空间的结构相似性，因此仅可忽略不计的修改。 FFHQ数据集上的广泛实验证明，我们提出的$ W $ ++空间显然比以前的$ w/w $+空间更可取。该代码可在https://github.com/anonsubm2021/transstylegan上公开提供。

Realistic image manipulation is challenging because it requires modifying the image appearance in a user-controlled way, while preserving the realism of the result. Unless the user has considerable artistic skill, it is easy to "fall off" the manifold of natural images while editing. In this paper, we propose to learn the natural image manifold directly from data using a generative adversarial neural network. We then define a class of image editing operations, and constrain their output to lie on that learned manifold at all times. The model automatically adjusts the output keeping all edits as realistic as possible. All our manipulations are expressed in terms of constrained optimization and are applied in near-real time. We evaluate our algorithm on the task of realistic photo manipulation of shape and color. The presented method can further be used for changing one image to look like the other, as well as generating novel imagery from scratch based on user's scribbles 1 .