制作生成模型3D感知桥梁2D图像空间和3D物理世界仍然挑战。最近尝试用神经辐射场(NERF)配备生成的对抗性网络(GAN),其将3D坐标映射到像素值,作为3D之前。然而,nerf中的隐式功能具有一个非常局部的接收领域,使得发电机难以意识到全局结构。与此同时,NERF建立在体积渲染上,这可能太昂贵,无法产生高分辨率结果,提高优化难度。为了减轻这两个问题,我们通过明确学习结构表示和纹理表示,向高保真3D感知图像综合提出了一种作为Volumegan称为Volumegan的新颖框架。我们首先学习一个特征卷来表示底层结构,然后使用类似NERF的模型转换为特征字段。特征字段进一步累积到作为纹理表示的2D特征图中,然后是用于外观合成的神经渲染器。这种设计使得能够独立控制形状和外观。广泛的数据集的大量实验表明,我们的方法比以前的方法实现了足够更高的图像质量和更好的3D控制。
translated by 谷歌翻译
We have witnessed rapid progress on 3D-aware image synthesis, leveraging recent advances in generative visual models and neural rendering. Existing approaches however fall short in two ways: first, they may lack an underlying 3D representation or rely on view-inconsistent rendering, hence synthesizing images that are not multi-view consistent; second, they often depend upon representation network architectures that are not expressive enough, and their results thus lack in image quality. We propose a novel generative model, named Periodic Implicit Generative Adversarial Networks (π-GAN or pi-GAN), for high-quality 3D-aware image synthesis. π-GAN leverages neural representations with periodic activation functions and volumetric rendering to represent scenes as view-consistent radiance fields. The proposed approach obtains state-of-the-art results for 3D-aware image synthesis with multiple real and synthetic datasets.
translated by 谷歌翻译
生成辐射场的进步推动了3D感知图像合成的边界。通过观察到3D对象应该从多个观点看起来真实的观察,这些方法将多视图约束引入正则化以从2D图像学习有效的3D辐射场。尽管有了进步,但由于形状彩色模糊,它们通常会缺少准确的3D形状,这限制了它们在下游任务中的适用性。在这项工作中,我们通过提出一种新的阴影引导的生成隐式模型来解决这种模糊性,能够学习持续改进的形状表示。我们的主要洞察力是,在不同的照明条件下,精确的3D形状还应产生逼真的渲染。通过明确地模拟照明和具有各种照明条件的阴影来实现这种多照明约束。通过将合成的图像馈送到鉴别器来导出梯度。为了补偿计算表面法线的额外计算负担,我们进一步通过表面跟踪设计了高效的体积渲染策略,将培训和推理时间分别将培训和推理时间减少了24%和48%。我们在多个数据集上的实验表明,该方法在捕获准确的基础3D形状时实现了光电型3D感知图像合成。我们展示了我们对现有方法的3D形重建的方法的改进性能,并展示了其对图像复兴的适用性。我们的代码将在https://github.com/xingangpan/shadegan发布。
translated by 谷歌翻译
While 2D generative adversarial networks have enabled high-resolution image synthesis, they largely lack an understanding of the 3D world and the image formation process. Thus, they do not provide precise control over camera viewpoint or object pose. To address this problem, several recent approaches leverage intermediate voxel-based representations in combination with differentiable rendering. However, existing methods either produce low image resolution or fall short in disentangling camera and scene properties, e.g., the object identity may vary with the viewpoint. In this paper, we propose a generative model for radiance fields which have recently proven successful for novel view synthesis of a single scene. In contrast to voxelbased representations, radiance fields are not confined to a coarse discretization of the 3D space, yet allow for disentangling camera and scene properties while degrading gracefully in the presence of reconstruction ambiguity. By introducing a multi-scale patch-based discriminator, we demonstrate synthesis of high-resolution images while training our model from unposed 2D images alone. We systematically analyze our approach on several challenging synthetic and real-world datasets. Our experiments reveal that radiance fields are a powerful representation for generative image synthesis, leading to 3D consistent models that render with high fidelity.
translated by 谷歌翻译
使用单视图2D照片仅集合,无监督的高质量多视图 - 一致的图像和3D形状一直是一个长期存在的挑战。现有的3D GAN是计算密集型的,也是没有3D-一致的近似;前者限制了所生成的图像的质量和分辨率,并且后者对多视图一致性和形状质量产生不利影响。在这项工作中,我们提高了3D GAN的计算效率和图像质量,而无需依赖这些近似。为此目的,我们介绍了一种表现力的混合明确隐式网络架构,与其他设计选择一起,不仅可以实时合成高分辨率多视图一致图像,而且还产生高质量的3D几何形状。通过解耦特征生成和神经渲染,我们的框架能够利用最先进的2D CNN生成器,例如Stylega2,并继承它们的效率和表现力。在其他实验中,我们展示了与FFHQ和AFHQ猫的最先进的3D感知合成。
translated by 谷歌翻译
Existing 3D-aware image synthesis approaches mainly focus on generating a single canonical object and show limited capacity in composing a complex scene containing a variety of objects. This work presents DisCoScene: a 3Daware generative model for high-quality and controllable scene synthesis. The key ingredient of our method is a very abstract object-level representation (i.e., 3D bounding boxes without semantic annotation) as the scene layout prior, which is simple to obtain, general to describe various scene contents, and yet informative to disentangle objects and background. Moreover, it serves as an intuitive user control for scene editing. Based on such a prior, the proposed model spatially disentangles the whole scene into object-centric generative radiance fields by learning on only 2D images with the global-local discrimination. Our model obtains the generation fidelity and editing flexibility of individual objects while being able to efficiently compose objects and the background into a complete scene. We demonstrate state-of-the-art performance on many scene datasets, including the challenging Waymo outdoor dataset. Project page: https://snap-research.github.io/discoscene/
translated by 谷歌翻译
利用图像生成模型的最新进展,现有的可控面图像合成方法能够生成具有某些可控性的高保真图像,例如控制生成的面部图像的形状,表达,纹理和姿势。但是,这些方法集中在2D图像生成模型上,这些模型容易在大表达和姿势变化下产生不一致的面部图像。在本文中,我们提出了一个新的基于NERF的条件3D面部合成框架,该框架可以通过从3D脸先进的3D面部施加显式3D条件来对生成的面部图像进行3D可控性。其核心是有条件的生成占用场(CGOF),可有效地强制生成的面部形状,以使其对给定的3D形态模型(3DMM)网格进行。为了准确控制合成图像的细粒3D面部形状,我们还将3D地标损耗以及体积翘曲损失纳入我们的合成算法中。实验验证了所提出的方法的有效性,该方法能够生成高保真的面部图像,并显示出比基于2D的最新可控制的面部合成方法更精确的3D可控性。在https://keqiangsun.github.io/projects/cgof上查找代码和演示。
translated by 谷歌翻译
3D感知图像生成建模旨在生成具有明确可控相机姿势的3D一致图像。最近的作品通过在非结构化的2D图像上培训神经辐射场(NERF)发电机,但仍然无法产生具有精细细节的高度现实图像。一个关键原因是体积表示学习的高记忆和计算成本大大限制了训练期间辐射集成的点样本的数量。不足的采样不仅限制了发电机的表现力,以处理细节细节,而且由于不稳定的蒙特卡罗采样引起的噪音,因此阻碍了有效的GaN训练。我们提出了一种新的方法,调节点采样和辐射场地学习在2D歧管上,体现为3D音量中的一组学习隐式表面。对于每个观看射线,我们计算射线表面交叉点并累积由网络产生的亮度。通过培训和渲染如此光辉的歧管,我们的发电机可以产生具有现实细节和强大的视觉3D一致性的高质量图像。
translated by 谷歌翻译
我们介绍了我们称呼STYLESDF的高分辨率,3D一致的图像和形状生成技术。我们的方法仅在单视图RGB数据上培训,并站在StyleGan2的肩部,用于图像生成,同时解决3D感知GANS中的两个主要挑战:1)RGB图像的高分辨率,视图 - 一致生成RGB图像,以及2)详细的3D形状。通过使用基于样式的2D发生器合并基于SDF的3D表示来实现这一目标。我们的3D隐式网络呈现出低分辨率的特征映射,其中基于样式的网络生成了View-Consive,1024x1024图像。值得注意的是,基于SDF的3D建模定义了详细的3D曲面,导致一致的卷渲染。在视觉和几何质量方面,我们的方法显示出更高的质量结果。
translated by 谷歌翻译
我们提出Volux-GaN,一种生成框架,以合成3D感知面孔的令人信服的回忆。我们的主要贡献是一种体积的HDRI可发感方法,可以沿着每个3D光线沿着任何所需的HDR环境图累计累积Albedo,漫射和镜面照明贡献。此外,我们展示了使用多个鉴别器监督图像分解过程的重要性。特别是,我们提出了一种数据增强技术,其利用单个图像肖像结合的最近的进步来强制实施一致的几何形状,反照镜,漫射和镜面组分。与其他生成框架的多个实验和比较展示了我们的模型是如何向光电型可致力于的3D生成模型前进的一步。
translated by 谷歌翻译
随着几个行业正在朝着建模大规模的3D虚拟世界迈进,因此需要根据3D内容的数量,质量和多样性来扩展的内容创建工具的需求变得显而易见。在我们的工作中,我们旨在训练Parterant 3D生成模型,以合成纹理网格,可以通过3D渲染引擎直接消耗,因此立即在下游应用中使用。 3D生成建模的先前工作要么缺少几何细节,因此在它们可以生成的网格拓扑中受到限制,通常不支持纹理,或者在合成过程中使用神经渲染器,这使得它们在常见的3D软件中使用。在这项工作中,我们介绍了GET3D,这是一种生成模型,该模型直接生成具有复杂拓扑,丰富几何细节和高保真纹理的显式纹理3D网格。我们在可区分的表面建模,可区分渲染以及2D生成对抗网络中桥接了最新成功,以从2D图像集合中训练我们的模型。 GET3D能够生成高质量的3D纹理网格,从汽车,椅子,动物,摩托车和人类角色到建筑物,对以前的方法进行了重大改进。
translated by 谷歌翻译
以前的纵向图像生成方法大致分为两类:2D GAN和3D感知的GAN。 2D GAN可以产生高保真肖像,但具有低视图一致性。 3D感知GaN方法可以维护查看一致性,但它们所生成的图像不是本地可编辑的。为了克服这些限制,我们提出了FENERF,一个可以生成查看一致和本地可编辑的纵向图像的3D感知生成器。我们的方法使用两个解耦潜码,以在具有共享几何体的空间对齐的3D卷中生成相应的面部语义和纹理。从这种底层3D表示中受益,FENERF可以联合渲染边界对齐的图像和语义掩码,并使用语义掩模通过GaN反转编辑3D音量。我们进一步示出了可以从广泛可用的单手套图像和语义面膜对中学习这种3D表示。此外,我们揭示了联合学习语义和纹理有助于产生更精细的几何形状。我们的实验表明FENERF在各种面部编辑任务中优于最先进的方法。
translated by 谷歌翻译
生成辐射田地的出现显着促进了3D感知图像合成的发展。辐射字段中的累积渲染过程使得这些生成模型更容易,因为渐变在整个音量上分布,但导致扩散的物体表面。与此同时,与Radiance Fields相比,占用表示可以本质地确保确定性表面。但是,如果我们直接向生成模型应用占用表示,在培训期间,它们只会在物体表面上接收稀疏梯度,并最终遭受收敛问题。在本文中,我们提出了一种基于生成的辐射场的新型模型的生成占用场(GOF),这些模型可以在不妨碍其训练收敛的情况下学习紧凑的物体表面。 GOF的关键介绍是从辐射字段中累积渲染到渲染的专用过渡,只有在学习的表面越来越准确的情况下,只有曲面点渲染。通过这种方式,GOF将两个表示的优点组合在统一的框架中。在实践中,通过逐渐将采样区域从整个体积逐渐缩小到表面周围的最小相邻区域,在GOF中实现了从辐射场和3月到占用表示的训练时间转换。通过对多个数据集的全面实验,我们证明了GOF可以合成具有3D一致性的高质量图像,并同时学习紧凑且光滑的物体表面。代码,模型和演示视频可在https://shedontsui.g​​ithub.io/projects/gof中获得
translated by 谷歌翻译
Generative models, as an important family of statistical modeling, target learning the observed data distribution via generating new instances. Along with the rise of neural networks, deep generative models, such as variational autoencoders (VAEs) and generative adversarial network (GANs), have made tremendous progress in 2D image synthesis. Recently, researchers switch their attentions from the 2D space to the 3D space considering that 3D data better aligns with our physical world and hence enjoys great potential in practice. However, unlike a 2D image, which owns an efficient representation (i.e., pixel grid) by nature, representing 3D data could face far more challenges. Concretely, we would expect an ideal 3D representation to be capable enough to model shapes and appearances in details, and to be highly efficient so as to model high-resolution data with fast speed and low memory cost. However, existing 3D representations, such as point clouds, meshes, and recent neural fields, usually fail to meet the above requirements simultaneously. In this survey, we make a thorough review of the development of 3D generation, including 3D shape generation and 3D-aware image synthesis, from the perspectives of both algorithms and more importantly representations. We hope that our discussion could help the community track the evolution of this field and further spark some innovative ideas to advance this challenging task.
translated by 谷歌翻译
Deep generative models allow for photorealistic image synthesis at high resolutions. But for many applications, this is not enough: content creation also needs to be controllable. While several recent works investigate how to disentangle underlying factors of variation in the data, most of them operate in 2D and hence ignore that our world is three-dimensional. Further, only few works consider the compositional nature of scenes. Our key hypothesis is that incorporating a compositional 3D scene representation into the generative model leads to more controllable image synthesis. Representing scenes as compositional generative neural feature fields allows us to disentangle one or multiple objects from the background as well as individual objects' shapes and appearances while learning from unstructured and unposed image collections without any additional supervision. Combining this scene representation with a neural rendering pipeline yields a fast and realistic image synthesis model. As evidenced by our experiments, our model is able to disentangle individual objects and allows for translating and rotating them in the scene as well as changing the camera pose.
translated by 谷歌翻译
最先进的3D感知生成模型依赖于基于坐标的MLP来参数化3D辐射场。在证明令人印象深刻的结果的同时,请查询每个沿每个射线样品的MLP,都会导致渲染缓慢。因此,现有方法通常会呈现低分辨率特征图,并通过UPSMPLING网络处理以获取最终图像。尽管有效,神经渲染通常纠缠于观点和内容,从而改变摄像头会导致几何或外观的不必要变化。在基于体素的新型视图合成中的最新结果中,我们研究了本文中稀疏体素电网表示的快速和3D一致生成建模的实用性。我们的结果表明,当将稀疏体素电网与渐进式生长,自由空间修剪和适当的正则化结合时,单层MLP确实可以被3D卷积代替。为了获得场景的紧凑表示并允许缩放到更高的体素分辨率,我们的模型将前景对象(以3D模型)从背景(以2D模型建模)中。与现有方法相反,我们的方法仅需要单个正向通行证来生成完整的3D场景。因此,它允许从任意观点呈现有效渲染,同时以高视觉保真度产生3D一致的结果。
translated by 谷歌翻译
StyleGAN has achieved great progress in 2D face reconstruction and semantic editing via image inversion and latent editing. While studies over extending 2D StyleGAN to 3D faces have emerged, a corresponding generic 3D GAN inversion framework is still missing, limiting the applications of 3D face reconstruction and semantic editing. In this paper, we study the challenging problem of 3D GAN inversion where a latent code is predicted given a single face image to faithfully recover its 3D shapes and detailed textures. The problem is ill-posed: innumerable compositions of shape and texture could be rendered to the current image. Furthermore, with the limited capacity of a global latent code, 2D inversion methods cannot preserve faithful shape and texture at the same time when applied to 3D models. To solve this problem, we devise an effective self-training scheme to constrain the learning of inversion. The learning is done efficiently without any real-world 2D-3D training pairs but proxy samples generated from a 3D GAN. In addition, apart from a global latent code that captures the coarse shape and texture information, we augment the generation network with a local branch, where pixel-aligned features are added to faithfully reconstruct face details. We further consider a new pipeline to perform 3D view-consistent editing. Extensive experiments show that our method outperforms state-of-the-art inversion methods in both shape and texture reconstruction quality. Code and data will be released.
translated by 谷歌翻译
与传统的头像创建管道相反,这是一个昂贵的过程,现代生成方法直接从照片中学习数据分布,而艺术的状态现在可以产生高度的照片现实图像。尽管大量作品试图扩展无条件的生成模型并达到一定程度的可控性,但要确保多视图一致性,尤其是在大型姿势中,仍然具有挑战性。在这项工作中,我们提出了一个3D肖像生成网络,该网络可产生3D一致的肖像,同时根据有关姿势,身份,表达和照明的语义参数可控。生成网络使用神经场景表示在3D中建模肖像,其生成以支持明确控制的参数面模型为指导。尽管可以通过将图像与部分不同的属性进行对比,但可以进一步增强潜在的分离,但在非面积区域(例如,在动画表达式)时,仍然存在明显的不一致。我们通过提出一种体积混合策略来解决此问题,在该策略中,我们通过将动态和静态辐射场融合在一起,形成一个复合输出,并从共同学习的语义场中分割了两个部分。我们的方法在广泛的实验中优于先前的艺术,在自由视点中观看时,在自然照明中产生了逼真的肖像。所提出的方法还证明了真实图像以及室外卡通面孔的概括能力,在实际应用中显示出巨大的希望。其他视频结果和代码将在项目网页上提供。
translated by 谷歌翻译
无监督的生成的虚拟人类具有各种外观和动画姿势对于创建3D人体化身和其他AR/VR应用非常重要。现有方法要么仅限于刚性对象建模,要么不生成,因此无法合成高质量的虚拟人类并使它们进行动画化。在这项工作中,我们提出了Avatargen,这是第一种不仅可以具有不同外观的非刚性人类产生的方法,而且还可以完全控制姿势和观点,同时仅需要2D图像进行训练。具体而言,它通过利用粗糙的人体模型作为代理将观察空间扭曲到规范空间下的标准头像,将最近的3D甘斯扩展到了人类的衣服。为了建模非刚性动力学,它引入了一个变形网络,以学习规范空间中的姿势依赖性变形。为了提高生成的人类化身的几何质量,它利用签名距离字段作为几何表示,从而可以从几何学学习上的身体模型中进行更直接的正则化。从这些设计中受益,我们的方法可以生成具有高质量外观和几何形状建模的动画人体化身,从而极大地表现了先前的3D gan。此外,它有能力用于许多应用,例如单视重构造,复活和文本引导的合成。代码和预培训模型将可用。
translated by 谷歌翻译
最近已经示出了从2D图像中提取隐式3D表示的生成神经辐射场(GNERF)模型,以产生代表刚性物体的现实图像,例如人面或汽车。然而,他们通常难以产生代表非刚性物体的高质量图像,例如人体,这对许多计算机图形应用具有很大的兴趣。本文提出了一种用于人类图像综合的3D感知语义导向生成模型(3D-SAGGA),其集成了GNERF和纹理发生器。前者学习人体的隐式3D表示,并输出一组2D语义分段掩模。后者将这些语义面部掩模转化为真实的图像,为人类的外观添加了逼真的纹理。如果不需要额外的3D信息,我们的模型可以使用照片现实可控生成学习3D人类表示。我们在Deepfashion DataSet上的实验表明,3D-SAGGAN显着优于最近的基线。
translated by 谷歌翻译