We propose an efficient algorithm to embed a given image into the latent space of StyleGAN. This embedding enables semantic image editing operations that can be applied to existing photographs. Taking the StyleGAN trained on the FFHQ dataset as an example, we show results for image morphing, style transfer, and expression transfer. Studying the results of the embedding algorithm provides valuable insights into the structure of the StyleGAN latent space. We propose a set of experiments to test what class of images can be embedded, how they are embedded, what latent space is suitable for embedding, and if the embedding is semantically meaningful.

Figure 1: (a) and (b): input images; (c): the "two-face" generated by naively copying the left half from (a) and the right half from (b); (d): the "two-face" generated by our Image2StyleGAN++ framework.

Figure 1. The proposed pixel2style2pixel framework can be used to solve a wide variety of image-to-image translation tasks. Here we show results of pSp on StyleGAN inversion, multi-modal conditional image synthesis, facial frontalization, inpainting and super-resolution.

Our goal with this survey is to provide an overview of the state of the art deep learning technologies for face generation and editing. We will cover popular latest architectures and discuss key ideas that make them work, such as inversion, latent representation, loss functions, training procedures, editing methods, and cross domain style transfer. We particularly focus on GAN-based architectures that have culminated in the StyleGAN approaches, which allow generation of high-quality face images and offer rich interfaces for controllable semantics editing and preserving photo quality. We aim to provide an entry point into the field for readers that have basic knowledge about the field of deep learning and are looking for an accessible introduction and overview.

最近在图像编辑中找到了生成的对抗网络（GANS）。但是，大多数基于GaN的图像编辑方法通常需要具有用于训练的语义分段注释的大规模数据集，只提供高级控制，或者仅在不同图像之间插入。在这里，我们提出了EditGan，一种用于高质量，高精度语义图像编辑的新方法，允许用户通过修改高度详细的部分分割面罩，例如，为汽车前灯绘制新掩模来编辑图像。编辑登上的GAN框架上建立联合模型图像及其语义分割，只需要少数标记的示例，使其成为编辑的可扩展工具。具体地，我们将图像嵌入GaN潜在空间中，并根据分割编辑执行条件潜代码优化，这有效地修改了图像。算优化优化，我们发现在实现编辑的潜在空间中找到编辑向量。该框架允许我们学习任意数量的编辑向量，然后可以直接应用于交互式速率的其他图像。我们通过实验表明，EditGan可以用前所未有的细节和自由来操纵图像，同时保留完整的图像质量。我们还可以轻松地组合多个编辑并执行超出EditGan训练数据的合理编辑。我们在各种图像类型上展示编辑，并定量优于标准编辑基准任务的几种先前编辑方法。

Recent work has shown that a variety of semantics emerge in the latent space of Generative Adversarial Networks (GANs) when being trained to synthesize images. However, it is difficult to use these learned semantics for real image editing. A common practice of feeding a real image to a trained GAN generator is to invert it back to a latent code. However, existing inversion methods typically focus on reconstructing the target image by pixel values yet fail to land the inverted code in the semantic domain of the original latent space. As a result, the reconstructed image cannot well support semantic editing through varying the inverted code. To solve this problem, we propose an in-domain GAN inversion approach, which not only faithfully reconstructs the input image but also ensures the inverted code to be semantically meaningful for editing. We first learn a novel domain-guided encoder to project a given image to the native latent space of GANs. We then propose domain-regularized optimization by involving the encoder as a regularizer to fine-tune the code produced by the encoder and better recover the target image. Extensive experiments suggest that our inversion method achieves satisfying real image reconstruction and more importantly facilitates various image editing tasks, significantly outperforming start-of-the-arts. 1

我们为一个拍摄域适应提供了一种新方法。我们方法的输入是训练的GaN，其可以在域B中产生域A和单个参考图像I_B的图像。所提出的算法可以将训练的GaN的任何输出从域A转换为域B.我们的主要优点有两个主要优点方法与当前现有技术相比：首先，我们的解决方案实现了更高的视觉质量，例如通过明显减少过度装箱。其次，我们的解决方案允许更多地控制域间隙的自由度，即图像I_B的哪些方面用于定义域B.从技术上讲，我们通过在预先训练的样式生成器上建立新方法作为GaN和A用于代表域间隙的预先训练的夹模型。我们提出了几种新的常规程序来控制域间隙，以优化预先训练的样式生成器的权重，以输出域B中的图像而不是域A.常规方法防止优化来自单个参考图像的太多属性。我们的结果表明，对现有技术的显着视觉改进以及突出了改进控制的多个应用程序。

We propose semantic region-adaptive normalization (SEAN), a simple but effective building block for Generative Adversarial Networks conditioned on segmentation masks that describe the semantic regions in the desired output image. Using SEAN normalization, we can build a network architecture that can control the style of each semantic region individually, e.g., we can specify one style reference image per region. SEAN is better suited to encode, transfer, and synthesize style than the best previous method in terms of reconstruction quality, variability, and visual quality. We evaluate SEAN on multiple datasets and report better quan-titative metrics (e.g. FID, PSNR) than the current state of the art. SEAN also pushes the frontier of interactive image editing. We can interactively edit images by changing segmentation masks or the style for any given region. We can also interpolate styles from two reference images per region. Code: https://github.com/ZPdesu/SEAN .

Recent 3D-aware GANs rely on volumetric rendering techniques to disentangle the pose and appearance of objects, de facto generating entire 3D volumes rather than single-view 2D images from a latent code. Complex image editing tasks can be performed in standard 2D-based GANs (e.g., StyleGAN models) as manipulation of latent dimensions. However, to the best of our knowledge, similar properties have only been partially explored for 3D-aware GAN models. This work aims to fill this gap by showing the limitations of existing methods and proposing LatentSwap3D, a model-agnostic approach designed to enable attribute editing in the latent space of pre-trained 3D-aware GANs. We first identify the most relevant dimensions in the latent space of the model controlling the targeted attribute by relying on the feature importance ranking of a random forest classifier. Then, to apply the transformation, we swap the top-K most relevant latent dimensions of the image being edited with an image exhibiting the desired attribute. Despite its simplicity, LatentSwap3D provides remarkable semantic edits in a disentangled manner and outperforms alternative approaches both qualitatively and quantitatively. We demonstrate our semantic edit approach on various 3D-aware generative models such as pi-GAN, GIRAFFE, StyleSDF, MVCGAN, EG3D and VolumeGAN, and on diverse datasets, such as FFHQ, AFHQ, Cats, MetFaces, and CompCars. The project page can be found: \url{https://enisimsar.github.io/latentswap3d/}.

由于GaN潜在空间的勘探和利用，近年来，现实世界的图像操纵实现了奇妙的进展。 GaN反演是该管道的第一步，旨在忠实地将真实图像映射到潜在代码。不幸的是，大多数现有的GaN反演方法都无法满足下面列出的三个要求中的至少一个：重建质量，可编辑性和快速推断。我们在本研究中提出了一种新的两阶段策略，同时适合所有要求。在第一阶段，我们训练编码器将输入图像映射到StyleGan2 $ \ Mathcal {W} $ - 空间，这被证明具有出色的可编辑性，但重建质量较低。在第二阶段，我们通过利用一系列HyperNetWorks来补充初始阶段的重建能力以在反转期间恢复缺失的信息。这两个步骤互相补充，由于Hypernetwork分支和由于$ \ Mathcal {W} $ - 空间中的反转，因此由于HyperNetwork分支和优异的可编辑性而相互作用。我们的方法完全是基于编码器的，导致极快的推断。关于两个具有挑战性的数据集的广泛实验证明了我们方法的优越性。

尽管使用StyleGan进行语义操纵的最新进展，但对真实面孔的语义编辑仍然具有挑战性。 $ W $空间与$ W $+空间之间的差距需要重建质量与编辑质量之间的不良权衡。为了解决这个问题，我们建议通过用基于注意的变压器代替Stylegan映射网络中的完全连接的层来扩展潜在空间。这种简单有效的技术将上述两个空间整合在一起，并将它们转换为一个名为$ W $ ++的新的潜在空间。我们的修改后的Stylegan保持了原始StyleGan的最新一代质量，并具有中等程度的多样性。但更重要的是，提议的$ W $ ++空间在重建质量和编辑质量方面都取得了卓越的性能。尽管有这些显着优势，但我们的$ W $ ++空间支持现有的反转算法和编辑方法，仅由于其与$ w/w $+空间的结构相似性，因此仅可忽略不计的修改。 FFHQ数据集上的广泛实验证明，我们提出的$ W $ ++空间显然比以前的$ w/w $+空间更可取。该代码可在https://github.com/anonsubm2021/transstylegan上公开提供。

We present a novel image inversion framework and a training pipeline to achieve high-fidelity image inversion with high-quality attribute editing. Inverting real images into StyleGAN's latent space is an extensively studied problem, yet the trade-off between the image reconstruction fidelity and image editing quality remains an open challenge. The low-rate latent spaces are limited in their expressiveness power for high-fidelity reconstruction. On the other hand, high-rate latent spaces result in degradation in editing quality. In this work, to achieve high-fidelity inversion, we learn residual features in higher latent codes that lower latent codes were not able to encode. This enables preserving image details in reconstruction. To achieve high-quality editing, we learn how to transform the residual features for adapting to manipulations in latent codes. We train the framework to extract residual features and transform them via a novel architecture pipeline and cycle consistency losses. We run extensive experiments and compare our method with state-of-the-art inversion methods. Qualitative metrics and visual comparisons show significant improvements. Code: https://github.com/hamzapehlivan/StyleRes

现有的神经样式传输方法需要参考样式图像来将样式图像的纹理信息传输到内容图像。然而，在许多实际情况中，用户可能没有参考样式图像，但仍然有兴趣通过想象它们来传输样式。为了处理此类应用程序，我们提出了一个新的框架，它可以实现样式转移`没有'风格图像，但仅使用所需风格的文本描述。使用预先训练的文本图像嵌入模型的剪辑，我们仅通过单个文本条件展示了内容图像样式的调制。具体而言，我们提出了一种针对现实纹理传输的多视图增强的修补程序文本图像匹配丢失。广泛的实验结果证实了具有反映语义查询文本的现实纹理的成功图像风格转移。

通过利用预熟gan的潜在空间，已经提出了许多最近的作品来进行面部图像编辑。但是，很少有尝试将它们直接应用于视频，因为1）他们不能保证时间一致性，2）他们的应用受到视频的处理速度的限制，3）他们无法准确编码面部运动和表达的细节。为此，我们提出了一个新颖的网络，将面部视频编码到Stylegan的潜在空间中，以进行语义面部视频操纵。基于视觉变压器，我们的网络重复了潜在向量的高分辨率部分，以实现时间一致性。为了捕捉微妙的面部运动和表情，我们设计了涉及稀疏面部地标和密集的3D脸部网眼的新颖损失。我们已经彻底评估了我们的方法，并成功证明了其对各种面部视频操作的应用。特别是，我们提出了一个新型网络，用于3D坐标系中的姿势/表达控制。定性和定量结果都表明，我们的方法可以显着优于现有的单图方法，同时实现实时（66 fps）速度。

We propose an alternative generator architecture for generative adversarial networks, borrowing from style transfer literature. The new architecture leads to an automatically learned, unsupervised separation of high-level attributes (e.g., pose and identity when trained on human faces) and stochastic variation in the generated images (e.g., freckles, hair), and it enables intuitive, scale-specific control of the synthesis. The new generator improves the state-of-the-art in terms of traditional distribution quality metrics, leads to demonstrably better interpolation properties, and also better disentangles the latent factors of variation. To quantify interpolation quality and disentanglement, we propose two new, automated methods that are applicable to any generator architecture. Finally, we introduce a new, highly varied and high-quality dataset of human faces.

现有的GAN倒置和编辑方法适用于具有干净背景的对齐物体，例如肖像和动物面孔，但通常会为更加困难的类别而苦苦挣扎，具有复杂的场景布局和物体遮挡，例如汽车，动物和室外图像。我们提出了一种新方法，以在gan的潜在空间（例如stylegan2）中倒转和编辑复杂的图像。我们的关键想法是用一系列层的集合探索反演，从而将反转过程适应图像的难度。我们学会预测不同图像段的“可逆性”，并将每个段投影到潜在层。更容易的区域可以倒入发电机潜在空间中的较早层，而更具挑战性的区域可以倒入更晚的特征空间。实验表明，与最新的复杂类别的方法相比，我们的方法获得了更好的反转结果，同时保持下游的编辑性。请参阅我们的项目页面，网址为https://www.cs.cmu.edu/~saminversion。

In this work, we are dedicated to text-guided image generation and propose a novel framework, i.e., CLIP2GAN, by leveraging CLIP model and StyleGAN. The key idea of our CLIP2GAN is to bridge the output feature embedding space of CLIP and the input latent space of StyleGAN, which is realized by introducing a mapping network. In the training stage, we encode an image with CLIP and map the output feature to a latent code, which is further used to reconstruct the image. In this way, the mapping network is optimized in a self-supervised learning way. In the inference stage, since CLIP can embed both image and text into a shared feature embedding space, we replace CLIP image encoder in the training architecture with CLIP text encoder, while keeping the following mapping network as well as StyleGAN model. As a result, we can flexibly input a text description to generate an image. Moreover, by simply adding mapped text features of an attribute to a mapped CLIP image feature, we can effectively edit the attribute to the image. Extensive experiments demonstrate the superior performance of our proposed CLIP2GAN compared to previous methods.

由于简单但有效的训练机制和出色的图像产生质量，生成的对抗网络（GAN）引起了极大的关注。具有生成照片现实的高分辨率（例如$ 1024 \ times1024 $）的能力，最近的GAN模型已大大缩小了生成的图像与真实图像之间的差距。因此，许多最近的作品表明，通过利用良好的潜在空间和博学的gan先验来利用预先训练的GAN模型的新兴兴趣。在本文中，我们简要回顾了从三个方面利用预先培训的大规模GAN模型的最新进展，即1）大规模生成对抗网络的培训，2）探索和理解预训练的GAN模型，以及预先培训的GAN模型，以及3）利用这些模型进行后续任务，例如图像恢复和编辑。有关相关方法和存储库的更多信息，请访问https://github.com/csmliu/pretretaining-gans。

Stone" "Mohawk hairstyle" "Without makeup" "Cute cat" "Lion" "Gothic church" * Equal contribution, ordered alphabetically. Code and video are available on https://github.com/orpatashnik/StyleCLIP