一击生成域Adaption旨在仅使用一个参考图像将一个预训练的发电机传输到一个新域中。但是,适用的生成器(i)要生成从预训练的生成器继承的多种图像,而(ii)(ii)忠实地获取参考图像的特定领域特定属性和样式,这仍然非常具有挑战性。在本文中,我们提出了一种新颖的单发性生成域适应方法,即Difa,用于多元化和忠实的适应。对于全球级别的适应,我们利用参考图像的剪辑嵌入与源图像的平均嵌入之间的差异来限制目标发生器。对于本地级别的适应,我们引入了一个细心的样式损失,该损失将每个适应图像的中间令牌与参考图像的相应令牌保持一致。为了促进多样化的生成,引入了选择性的跨域一致性,以选择和保留域共享属性,以编辑潜在的$ \ MATHCAL {W}+$ $空间来继承预训练的生成器的多样性。广泛的实验表明,我们的方法在定量和定性上都优于最先进的实验,尤其是对于大域间隙的情况。此外,我们的DIFA可以轻松地扩展到零击生成域的适应性,并具有吸引力的结果。代码可从https://github.com/1170300521/difa获得。
translated by 谷歌翻译
可以训练生成模型,以从特定域中生成图像,仅由文本提示引导,而不看到任何图像?换句话说:可以将图像生成器“盲目地训练”吗?利用大规模对比语言图像预训练(CLIP)模型的语义力量,我们提出了一种文本驱动方法,允许将生成模型转移到新域,而无需收集单个图像。我们展示通过自然语言提示和几分钟的培训,我们的方法可以通过各种风格和形状的多种域调整发电机。值得注意的是,许多这些修改难以与现有方法达到困难或完全不可能。我们在广泛的域中进行了广泛的实验和比较。这些证明了我们方法的有效性,并表明我们的移动模型保持了对下游任务吸引的生成模型的潜在空间属性。
translated by 谷歌翻译
我们为一个拍摄域适应提供了一种新方法。我们方法的输入是训练的GaN,其可以在域B中产生域A和单个参考图像I_B的图像。所提出的算法可以将训练的GaN的任何输出从域A转换为域B.我们的主要优点有两个主要优点方法与当前现有技术相比:首先,我们的解决方案实现了更高的视觉质量,例如通过明显减少过度装箱。其次,我们的解决方案允许更多地控制域间隙的自由度,即图像I_B的哪些方面用于定义域B.从技术上讲,我们通过在预先训练的样式生成器上建立新方法作为GaN和A用于代表域间隙的预先训练的夹模型。我们提出了几种新的常规程序来控制域间隙,以优化预先训练的样式生成器的权重,以输出域B中的图像而不是域A.常规方法防止优化来自单个参考图像的太多属性。我们的结果表明,对现有技术的显着视觉改进以及突出了改进控制的多个应用程序。
translated by 谷歌翻译
这项工作旨在将在一个图像域上预先训练的生成的对抗网络(GaN)转移到新域名,其仅仅是只有一个目标图像。主要挑战是,在有限的监督下,综合照片现实和高度多样化的图像非常困难,同时获取目标的代表性。不同于采用Vanilla微调策略的现有方法,我们分别将两个轻量级模块导入发电机和鉴别器。具体地,我们将属性适配器引入发电机中冻结其原始参数,通过该参数,它可以通过其重复利用现有知识,因此保持合成质量和多样性。然后,我们用一个属性分类器装备了学习良好的鉴别器骨干,以确保生成器从引用中捕获相应的字符。此外,考虑到培训数据的多样性差(即,只有一个图像),我们建议在培训过程中建议在生成域中的多样性限制,减轻优化难度。我们的方法在各种环境下提出了吸引力的结果,基本上超越了最先进的替代方案,特别是在合成多样性方面。明显的是,我们的方法即使具有大域间隙,并且在几分钟内为每个实验提供鲁棒地收敛。
translated by 谷歌翻译
生成对抗网络(GAN)的适应旨在将预训练的GAN转移到具有有限培训数据的给定领域。在本文中,我们专注于单次案例,这在以前的作品中更具挑战性,很少探索。我们认为,从源域到目标域的适应性可以分为两个部分:全球样式(如纹理和颜色)的转移,以及不属于源域的新实体的出现。虽然先前的作品主要关注样式转移,但我们提出了一个新颖而简洁的框架\ footNote {\ url {https://github.com/thevoidname/generalized-onerized-one-one-shot-gan-adaption}},以解决\ textit {对样式和实体传输的一般性单发适应性}任务,其中提供了参考图像及其二进制实体掩码。我们的核心目标是通过切成薄片的瓦斯坦距离来限制参考文献和合成的内部分布之间的差距。为了更好地实现这一目标,首先使用样式固定来大致获得模范样式,并将辅助网络引入原始生成器以删除实体和样式传输。此外,为了实现跨域的对应关系,我们提出了变异的拉普拉斯正则化以限制适应性发生器的平滑度。定量和定性实验都证明了我们方法在各种情况下的有效性。
translated by 谷歌翻译
Domain adaptation of GANs is a problem of fine-tuning the state-of-the-art GAN models (e.g. StyleGAN) pretrained on a large dataset to a specific domain with few samples (e.g. painting faces, sketches, etc.). While there are a great number of methods that tackle this problem in different ways there are still many important questions that remain unanswered. In this paper, we provide a systematic and in-depth analysis of the domain adaptation problem of GANs, focusing on the StyleGAN model. First, we perform a detailed exploration of the most important parts of StyleGAN that are responsible for adapting the generator to a new domain depending on the similarity between the source and target domains. In particular, we show that affine layers of StyleGAN can be sufficient for fine-tuning to similar domains. Second, inspired by these findings, we investigate StyleSpace to utilize it for domain adaptation. We show that there exist directions in the StyleSpace that can adapt StyleGAN to new domains. Further, we examine these directions and discover their many surprising properties. Finally, we leverage our analysis and findings to deliver practical improvements and applications in such standard tasks as image-to-image translation and cross-domain morphing.
translated by 谷歌翻译
由于发型的复杂性和美味,编辑发型是独一无二的,而且具有挑战性。尽管最近的方法显着改善了头发的细节,但是当源图像的姿势与目标头发图像的姿势大不相同时,这些模型通常会产生不良的输出,从而限制了其真实世界的应用。发型是一种姿势不变的发型转移模型,可以减轻这种限制,但在保留精致的头发质地方面仍然表现出不令人满意的质量。为了解决这些局限性,我们提出了配备潜在优化和新呈现的局部匹配损失的高性能姿势不变的发型转移模型。在stylegan2潜在空间中,我们首先探索目标头发的姿势对准的潜在代码,并根据本地风格匹配保留了详细纹理。然后,我们的模型对源的遮挡构成了对齐的目标头发的遮挡,并将两个图像混合在一起以产生最终输出。实验结果表明,我们的模型在在较大的姿势差异和保留局部发型纹理下转移发型方面具有优势。
translated by 谷歌翻译
In this work, we are dedicated to text-guided image generation and propose a novel framework, i.e., CLIP2GAN, by leveraging CLIP model and StyleGAN. The key idea of our CLIP2GAN is to bridge the output feature embedding space of CLIP and the input latent space of StyleGAN, which is realized by introducing a mapping network. In the training stage, we encode an image with CLIP and map the output feature to a latent code, which is further used to reconstruct the image. In this way, the mapping network is optimized in a self-supervised learning way. In the inference stage, since CLIP can embed both image and text into a shared feature embedding space, we replace CLIP image encoder in the training architecture with CLIP text encoder, while keeping the following mapping network as well as StyleGAN model. As a result, we can flexibly input a text description to generate an image. Moreover, by simply adding mapped text features of an attribute to a mapped CLIP image feature, we can effectively edit the attribute to the image. Extensive experiments demonstrate the superior performance of our proposed CLIP2GAN compared to previous methods.
translated by 谷歌翻译
本文介绍了DCT-NET,这是一种新颖的图像翻译体系结构,可用于几张肖像风格。给定有限的样式示例($ \ sim $ 100),新的体系结构可以产生高质量的样式转移结果,具有先进的能力,可以合成高保真内容和强大的一般性来处理复杂的场景(例如,遮挡和配件)。此外,它可以通过一个由部分观察(即风格化的头)训练的优雅评估网络启用全身图像翻译。几乎没有基于学习的样式转移是具有挑战性的,因为由于仅由少数几个培训示例形成的偏见分布,学到的模型很容易在目标域中过度拟合。本文旨在通过采用“首先校准,稍后翻译”的关键思想来应对挑战,并以本地注重的翻译探索增强的全球结构。具体而言,所提出的DCT-NET由三个模块组成:一个内容适配器从源照片借用功能的先验来校准目标样本的内容分布;使用仿射变换来释放空间语义约束的几何扩展模块;以及通过校准分布产生的样品的质地翻译模块学习细粒的转换。实验结果证明了所提出的方法在头部风格化方面具有优势及其对具有自适应变形的完整图像翻译的有效性。
translated by 谷歌翻译
The introduction of high-quality image generation models, particularly the StyleGAN family, provides a powerful tool to synthesize and manipulate images. However, existing models are built upon high-quality (HQ) data as desired outputs, making them unfit for in-the-wild low-quality (LQ) images, which are common inputs for manipulation. In this work, we bridge this gap by proposing a novel GAN structure that allows for generating images with controllable quality. The network can synthesize various image degradation and restore the sharp image via a quality control code. Our proposed QC-StyleGAN can directly edit LQ images without altering their quality by applying GAN inversion and manipulation techniques. It also provides for free an image restoration solution that can handle various degradations, including noise, blur, compression artifacts, and their mixtures. Finally, we demonstrate numerous other applications such as image degradation synthesis, transfer, and interpolation.
translated by 谷歌翻译
Recent 3D-aware GANs rely on volumetric rendering techniques to disentangle the pose and appearance of objects, de facto generating entire 3D volumes rather than single-view 2D images from a latent code. Complex image editing tasks can be performed in standard 2D-based GANs (e.g., StyleGAN models) as manipulation of latent dimensions. However, to the best of our knowledge, similar properties have only been partially explored for 3D-aware GAN models. This work aims to fill this gap by showing the limitations of existing methods and proposing LatentSwap3D, a model-agnostic approach designed to enable attribute editing in the latent space of pre-trained 3D-aware GANs. We first identify the most relevant dimensions in the latent space of the model controlling the targeted attribute by relying on the feature importance ranking of a random forest classifier. Then, to apply the transformation, we swap the top-K most relevant latent dimensions of the image being edited with an image exhibiting the desired attribute. Despite its simplicity, LatentSwap3D provides remarkable semantic edits in a disentangled manner and outperforms alternative approaches both qualitatively and quantitatively. We demonstrate our semantic edit approach on various 3D-aware generative models such as pi-GAN, GIRAFFE, StyleSDF, MVCGAN, EG3D and VolumeGAN, and on diverse datasets, such as FFHQ, AFHQ, Cats, MetFaces, and CompCars. The project page can be found: \url{https://enisimsar.github.io/latentswap3d/}.
translated by 谷歌翻译
由于其语义上的理解和用户友好的可控性,通过三维引导,通过三维引导的面部图像操纵已广泛应用于各种交互式场景。然而,现有的基于3D形式模型的操作方法不可直接适用于域名面,例如非黑色素化绘画,卡通肖像,甚至是动物,主要是由于构建每个模型的强大困难具体面部域。为了克服这一挑战,据我们所知,我们建议使用人为3DMM操纵任意域名的第一种方法。这是通过两个主要步骤实现的:1)从3DMM参数解开映射到潜在的STYLEGO2的潜在空间嵌入,可确保每个语义属性的解除响应和精确的控制; 2)通过实施一致的潜空间嵌入,桥接域差异并使人类3DMM适用于域外面的人类3DMM。实验和比较展示了我们高质量的语义操作方法在各种面部域中的优越性,所有主要3D面部属性可控姿势,表达,形状,反照镜和照明。此外,我们开发了直观的编辑界面,以支持用户友好的控制和即时反馈。我们的项目页面是https://cassiepython.github.io/cddfm3d/index.html
translated by 谷歌翻译
在本文中,我们解决了神经面部重演的问题,鉴于一对源和目标面部图像,我们需要通过将目标的姿势(定义为头部姿势及其面部表情定义)通过同时保留源的身份特征(例如面部形状,发型等),即使在源头和目标面属于不同身份的挑战性情况下也是如此。在此过程中,我们解决了最先进作品的一些局限在推理期间标记的数据以及c)它们不保留大型头部姿势变化中的身份。更具体地说,我们提出了一个框架,该框架使用未配对的随机生成的面部图像学会通过合并最近引入的样式空间$ \ Mathcal $ \ Mathcal {S} $ of Stylegan2的姿势,以将面部的身份特征从其姿势中解脱出来表现出显着的分解特性。通过利用这一点,我们学会使用3D模型的监督成功地混合了一对源和目标样式代码。随后用于重新制定的最终潜在代码由仅与源的面部姿势相对应的潜在单位和仅与源身份相对应的单位组成,从而显着改善了与最近的状态性能相比的重新制定性能。艺术方法。与艺术的状态相比,我们定量和定性地表明,即使在极端的姿势变化下,提出的方法也会产生更高的质量结果。最后,我们通过首先将它们嵌入预告片发电机的潜在空间来报告实际图像。我们在:https://github.com/stelabou/stylemask上公开提供代码和预估计的模型
translated by 谷歌翻译
利用Stylegan的表现力及其分离的潜在代码,现有方法可以实现对不同视觉属性的现实编辑,例如年龄和面部图像的性别。出现了一个有趣而又具有挑战性的问题:生成模型能否针对他们博学的先验进行反事实编辑?由于自然数据集中缺乏反事实样本,我们以文本驱动的方式研究了这个问题,并具有对比语言图像预言(剪辑),这些(剪辑)甚至可以为各种反事实概念提供丰富的语义知识。与内域操作不同,反事实操作需要更全面地剥削夹包含的语义知识,以及对编辑方向的更微妙的处理,以避免被卡在局部最低或不需要的编辑中。为此,我们设计了一种新颖的对比损失,该损失利用了预定义的夹子空间方向,从不同的角度将编辑指向所需的方向。此外,我们设计了一个简单而有效的方案,该方案将(目标文本)明确映射到潜在空间,并将其与潜在代码融合在一起,以进行有效的潜在代码优化和准确的编辑。广泛的实验表明,我们的设计在乘坐各种反事实概念的目标文本驾驶时,可以实现准确,现实的编辑。
translated by 谷歌翻译
Our goal with this survey is to provide an overview of the state of the art deep learning technologies for face generation and editing. We will cover popular latest architectures and discuss key ideas that make them work, such as inversion, latent representation, loss functions, training procedures, editing methods, and cross domain style transfer. We particularly focus on GAN-based architectures that have culminated in the StyleGAN approaches, which allow generation of high-quality face images and offer rich interfaces for controllable semantics editing and preserving photo quality. We aim to provide an entry point into the field for readers that have basic knowledge about the field of deep learning and are looking for an accessible introduction and overview.
translated by 谷歌翻译
现在,使用最近的生成对抗网络(GAN)可以使用高现实主义的不受约束图像产生。但是,用给定的一组属性生成图像非常具有挑战性。最近的方法使用基于样式的GAN模型来执行图像编辑,通过利用发电机层中存在的语义层次结构。我们提出了一些基于潜在的属性操纵和编辑(火焰),这是一个简单而有效的框架,可通过潜在空间操纵执行高度控制的图像编辑。具体而言,我们估计了控制生成图像中语义属性的潜在空间(预训练样式的)中的线性方向。与以前的方法相反,这些方法依赖于大规模属性标记的数据集或属性分类器,而火焰则使用一些策划的图像对的最小监督来估算删除的编辑指示。火焰可以在保留身份的同时,在各种图像集上同时进行高精度和顺序编辑。此外,我们提出了一项新颖的属性样式操纵任务,以生成各种样式的眼镜和头发等属性。我们首先编码相同身份的一组合成图像,但在潜在空间中具有不同的属性样式,以估计属性样式歧管。从该歧管中采样新的潜在将导致生成图像中的新属性样式。我们提出了一种新颖的抽样方法,以从歧管中采样潜在的样品,使我们能够生成各种属性样式,而不是训练集中存在的样式。火焰可以以分离的方式生成多种属性样式。我们通过广泛的定性和定量比较来说明火焰与先前的图像编辑方法相对于先前的图像编辑方法的卓越性能。火焰在多个数据集(例如汽车和教堂)上也很好地概括了。
translated by 谷歌翻译
Recent work has shown that a variety of semantics emerge in the latent space of Generative Adversarial Networks (GANs) when being trained to synthesize images. However, it is difficult to use these learned semantics for real image editing. A common practice of feeding a real image to a trained GAN generator is to invert it back to a latent code. However, existing inversion methods typically focus on reconstructing the target image by pixel values yet fail to land the inverted code in the semantic domain of the original latent space. As a result, the reconstructed image cannot well support semantic editing through varying the inverted code. To solve this problem, we propose an in-domain GAN inversion approach, which not only faithfully reconstructs the input image but also ensures the inverted code to be semantically meaningful for editing. We first learn a novel domain-guided encoder to project a given image to the native latent space of GANs. We then propose domain-regularized optimization by involving the encoder as a regularizer to fine-tune the code produced by the encoder and better recover the target image. Extensive experiments suggest that our inversion method achieves satisfying real image reconstruction and more importantly facilitates various image editing tasks, significantly outperforming start-of-the-arts. 1
translated by 谷歌翻译
最近,大型预磨损模型(例如,BERT,STYLEGAN,CLIP)在其域内的各种下游任务中表现出很好的知识转移和泛化能力。在这些努力的启发中,在本文中,我们提出了一个统一模型,用于开放域图像编辑,重点是开放式域图像的颜色和音调调整,同时保持原始内容和结构。我们的模型了解许多现有照片编辑软件中使用的操作空间(例如,对比度,亮度,颜色曲线)更具语义,直观,易于操作的统一编辑空间。我们的模型属于图像到图像转换框架,由图像编码器和解码器组成,并且在图像之前和图像的成对上培训以产生多模式输出。我们认为,通过将图像对反馈到学习编辑空间的潜在代码中,我们的模型可以利用各种下游编辑任务,例如语言引导图像编辑,个性化编辑,编辑式聚类,检索等。我们广泛地研究实验中编辑空间的独特属性,并在上述任务上展示了卓越的性能。
translated by 谷歌翻译
In this work, we propose TediGAN, a novel framework for multi-modal image generation and manipulation with textual descriptions. The proposed method consists of three components: StyleGAN inversion module, visual-linguistic similarity learning, and instance-level optimization. The inversion module maps real images to the latent space of a well-trained StyleGAN. The visual-linguistic similarity learns the text-image matching by mapping the image and text into a common embedding space. The instancelevel optimization is for identity preservation in manipulation. Our model can produce diverse and high-quality images with an unprecedented resolution at 1024 2 . Using a control mechanism based on style-mixing, our Tedi-GAN inherently supports image synthesis with multi-modal inputs, such as sketches or semantic labels, with or without instance guidance. To facilitate text-guided multimodal synthesis, we propose the Multi-Modal CelebA-HQ, a large-scale dataset consisting of real face images and corresponding semantic segmentation map, sketch, and textual descriptions. Extensive experiments on the introduced dataset demonstrate the superior performance of our proposed method. Code and data are available at https://github.com/weihaox/TediGAN.
translated by 谷歌翻译