在面孔和机构的3D生成模型中学习解除一致,可解释和结构化的潜在代表仍然是一个开放的问题。当需要对身份特征的控制时,问题特别严重。在本文中,我们提出了一种直观但有效的自我监督方法来训练3D形变形自动化器(VAE),鼓励身份特征的解开潜在表示。通过在不同形状上交换任意特征来造成迷你批处理允许定义利用潜在表示中已知差异和相似性的损耗功能。在3D网眼上进行的实验结果表明,最先进的潜在解剖学方法无法解散面部和身体的身份特征。我们所提出的方法适当地解耦了这些特征的产生,同时保持了良好的表示和重建能力。
translated by 谷歌翻译
Learned 3D representations of human faces are useful for computer vision problems such as 3D face tracking and reconstruction from images, as well as graphics applications such as character generation and animation. Traditional models learn a latent representation of a face using linear subspaces or higher-order tensor generalizations. Due to this linearity, they can not capture extreme deformations and nonlinear expressions. To address this, we introduce a versatile model that learns a non-linear representation of a face using spectral convolutions on a mesh surface. We introduce mesh sampling operations that enable a hierarchical mesh representation that captures non-linear variations in shape and expression at multiple scales within the model. In a variational setting, our model samples diverse realistic 3D faces from a multivariate Gaussian distribution. Our training data consists of 20,466 meshes of extreme expressions captured over 12 different subjects. Despite limited training data, our trained model outperforms state-of-the-art face models with 50% lower reconstruction error, while using 75% fewer parameters. We show that, replacing the expression space of an existing state-of-theart face model with our model, achieves a lower reconstruction error. Our data, model and code are available at http://coma.is.tue.mpg.de/.
translated by 谷歌翻译
在本文中,我们展示了Facetunegan,一种新的3D面部模型表示分解和编码面部身份和面部表情。我们提出了对图像到图像翻译网络的第一次适应,该图像已经成功地用于2D域,到3D面几何。利用最近释放的大面扫描数据库,神经网络已经过培训,以便与面部更好的了解,使面部表情转移和中和富有效应面的变异因素。具体而言,我们设计了一种适应基础架构的对抗架构,并使用Spiralnet ++进行卷积和采样操作。使用两个公共数据集(FACESCAPE和COMA),Facetunegan具有比最先进的技术更好的身份分解和面部中和。它还通过预测较近地面真实数据的闪烁形状并且由于源极和目标之间的面部形态过于不同的面部形态而越来越多的不期望的伪像来优异。
translated by 谷歌翻译
Generative models, as an important family of statistical modeling, target learning the observed data distribution via generating new instances. Along with the rise of neural networks, deep generative models, such as variational autoencoders (VAEs) and generative adversarial network (GANs), have made tremendous progress in 2D image synthesis. Recently, researchers switch their attentions from the 2D space to the 3D space considering that 3D data better aligns with our physical world and hence enjoys great potential in practice. However, unlike a 2D image, which owns an efficient representation (i.e., pixel grid) by nature, representing 3D data could face far more challenges. Concretely, we would expect an ideal 3D representation to be capable enough to model shapes and appearances in details, and to be highly efficient so as to model high-resolution data with fast speed and low memory cost. However, existing 3D representations, such as point clouds, meshes, and recent neural fields, usually fail to meet the above requirements simultaneously. In this survey, we make a thorough review of the development of 3D generation, including 3D shape generation and 3D-aware image synthesis, from the perspectives of both algorithms and more importantly representations. We hope that our discussion could help the community track the evolution of this field and further spark some innovative ideas to advance this challenging task.
translated by 谷歌翻译
Recent approaches to drape garments quickly over arbitrary human bodies leverage self-supervision to eliminate the need for large training sets. However, they are designed to train one network per clothing item, which severely limits their generalization abilities. In our work, we rely on self-supervision to train a single network to drape multiple garments. This is achieved by predicting a 3D deformation field conditioned on the latent codes of a generative network, which models garments as unsigned distance fields. Our pipeline can generate and drape previously unseen garments of any topology, whose shape can be edited by manipulating their latent codes. Being fully differentiable, our formulation makes it possible to recover accurate 3D models of garments from partial observations -- images or 3D scans -- via gradient descent. Our code will be made publicly available.
translated by 谷歌翻译
许多天然形状的大部分特征特征集中在太空中的几个地区。例如,人类和动物具有独特的头形,而椅子和飞机等无机物体则由具有特定几何特征的良好定位功能部件制成。通常,这些特征是密切相关的 - 四足动物中面部特征的修改应引起身体结构的变化。但是,在形状建模应用中,这些类型的编辑是最难的编辑。他们需要高精度,但也需要全球对整个形状的认识。即使在深度学习时代,获得满足此类要求的可操作表征也是一个开放的问题,构成了重大限制。在这项工作中,我们通过将数据驱动的模型定义为线性操作员(网状拉普拉斯的变体)来解决此问题,该模型的光谱捕获了手头形状的全局和局部几何特性。对这些光谱的修改被转化为相应表面的语义有效变形。通过明确将全局与本地表面特征分离,我们的管道允许执行本地编辑,同时保持全局风格的连贯性。我们凭经验证明了我们的基于学习的模型如何推广以塑造在培训时间看不到的表示,并且我们系统地分析了本地运营商在各种形状类别上的不同选择。
translated by 谷歌翻译
皮肤多人线性(SMPL)模型可以通过将姿势和形状参数映射到体网格来代表人体。已经示出了通过不同的学习模型方便推断3D人类姿势和形状。但是,并非所有的姿势和形状参数值都会产生物理合理的甚至现实的身体网格。换句话说,SMPL受到不受限制的,因此可以通过直接优化其参数来重建从图像的人类或通过从图像学习映射到这些参数来导致从图像中的人类来实现无效的结果。在本文中,我们学习之前将SMPL参数限制为通过对抗性培训产生现实姿势的值。我们表明,我们的学习了先前涵盖了实际数据分布的多样性,便于从2D关卡点进行3D重建的优化,并在用于从图像回归时产生更好的姿势估计。我们发现基于球面分布的先前获得了最佳效果。此外,在所有这些任务中,它优于基于最先进的VAE的方法来限制SMPL参数。
translated by 谷歌翻译
We present PhoMoH, a neural network methodology to construct generative models of photorealistic 3D geometry and appearance of human heads including hair, beards, clothing and accessories. In contrast to prior work, PhoMoH models the human head using neural fields, thus supporting complex topology. Instead of learning a head model from scratch, we propose to augment an existing expressive head model with new features. Concretely, we learn a highly detailed geometry network layered on top of a mid-resolution head model together with a detailed, local geometry-aware, and disentangled color field. Our proposed architecture allows us to learn photorealistic human head models from relatively little data. The learned generative geometry and appearance networks can be sampled individually and allow the creation of diverse and realistic human heads. Extensive experiments validate our method qualitatively and across different metrics.
translated by 谷歌翻译
Three-dimensional geometric data offer an excellent domain for studying representation learning and generative modeling. In this paper, we look at geometric data represented as point clouds. We introduce a deep AutoEncoder (AE) network with state-of-the-art reconstruction quality and generalization ability. The learned representations outperform existing methods on 3D recognition tasks and enable shape editing via simple algebraic manipulations, such as semantic part editing, shape analogies and shape interpolation, as well as shape completion. We perform a thorough study of different generative models including GANs operating on the raw point clouds, significantly improved GANs trained in the fixed latent space of our AEs, and Gaussian Mixture Models (GMMs). To quantitatively evaluate generative models we introduce measures of sample fidelity and diversity based on matchings between sets of point clouds. Interestingly, our evaluation of generalization, fidelity and diversity reveals that GMMs trained in the latent space of our AEs yield the best results overall.
translated by 谷歌翻译
随着几个行业正在朝着建模大规模的3D虚拟世界迈进,因此需要根据3D内容的数量,质量和多样性来扩展的内容创建工具的需求变得显而易见。在我们的工作中,我们旨在训练Parterant 3D生成模型,以合成纹理网格,可以通过3D渲染引擎直接消耗,因此立即在下游应用中使用。 3D生成建模的先前工作要么缺少几何细节,因此在它们可以生成的网格拓扑中受到限制,通常不支持纹理,或者在合成过程中使用神经渲染器,这使得它们在常见的3D软件中使用。在这项工作中,我们介绍了GET3D,这是一种生成模型,该模型直接生成具有复杂拓扑,丰富几何细节和高保真纹理的显式纹理3D网格。我们在可区分的表面建模,可区分渲染以及2D生成对抗网络中桥接了最新成功,以从2D图像集合中训练我们的模型。 GET3D能够生成高质量的3D纹理网格,从汽车,椅子,动物,摩托车和人类角色到建筑物,对以前的方法进行了重大改进。
translated by 谷歌翻译
我们认为人类变形转移问题,目标是在不同角色之间的零件姿势。解决此问题的传统方法需要清晰的姿势定义,并使用此定义在字符之间传输姿势。在这项工作中,我们采取了不同的方法,将角色的身份转换为新的身份,而无需修改角色的姿势。这提供了不必在3D人类姿势之间定义等效性的优点,这在姿势往往会根据执行它们的角色的身份而变化并不简单,并且由于它们的含义是高度上下文的。为了实现变形转移,我们提出了一种神经编码器 - 解码器架构,其中仅编码身份信息以及解码器在姿势上调节的位置。我们使用姿势独立表示,例如等距 - 不变形状特征,以表示身份特征。我们的模型使用这些功能来监督从变形姿势的偏移预测到转移结果。我们通过实验展示了我们的方法优于最先进的方法,定量和定性,并且更好地推广在训练期间没有看到。我们还介绍了一个微调步骤,可以为极端身份获得竞争力的结果,并允许转移简单的衣服。
translated by 谷歌翻译
在这项工作中,我们解决了4D面部表情生成的问题。通常,通过对中性3D面动画来达到表达峰,然后回到中立状态来解决这一问题。但是,在现实世界中,人们表现出更复杂的表情,并从一个表达式转换为另一种表达。因此,我们提出了一个新模型,该模型在不同表达式之间产生过渡,并综合了长长的4D表达式。这涉及三个子问题:(i)建模表达式的时间动力学,(ii)它们之间的学习过渡,以及(iii)变形通用网格。我们建议使用一组3D地标的运动编码表达式的时间演变,我们学会通过训练一个具有歧管值的gan(Motion3dgan)来生成。为了允许生成组成的表达式,该模型接受两个编码起始和结尾表达式的标签。网格的最终顺序是由稀疏的2块网格解码器(S2D-DEC)生成的,该解码器将地标位移映射到已知网格拓扑的密集,每位vertex位移。通过明确处理运动轨迹,该模型完全独立于身份。五个公共数据集的广泛实验表明,我们提出的方法在以前的解决方案方面带来了重大改进,同时保留了良好的概括以看不见数据。
translated by 谷歌翻译
The underlying dynamics and patterns of 3D surface meshes deforming over time can be discovered by unsupervised learning, especially autoencoders, which calculate low-dimensional embeddings of the surfaces. To study the deformation patterns of unseen shapes by transfer learning, we want to train an autoencoder that can analyze new surface meshes without training a new network. Here, most state-of-the-art autoencoders cannot handle meshes of different connectivity and therefore have limited to no generalization capacities to new meshes. Also, reconstruction errors strongly increase in comparison to the errors for the training shapes. To address this, we propose a novel spectral CoSMA (Convolutional Semi-Regular Mesh Autoencoder) network. This patch-based approach is combined with a surface-aware training. It reconstructs surfaces not presented during training and generalizes the deformation behavior of the surfaces' patches. The novel approach reconstructs unseen meshes from different datasets in superior quality compared to state-of-the-art autoencoders that have been trained on these shapes. Our transfer learning errors on unseen shapes are 40% lower than those from models learned directly on the data. Furthermore, baseline autoencoders detect deformation patterns of unseen mesh sequences only for the whole shape. In contrast, due to the employed regional patches and stable reconstruction quality, we can localize where on the surfaces these deformation patterns manifest.
translated by 谷歌翻译
甚至在没有受限,监督的情况下,也提出了甚至在没有受限或有限的情况下学习普遍陈述的方法。使用适度数量的数据可以微调新的目标任务,或者直接在相应任务中实现显着性能的无奈域中使用的良好普遍表示。这种缓解数据和注释要求为计算机愿景和医疗保健的应用提供了诱人的前景。在本辅导纸上,我们激励了对解散的陈述,目前关键理论和详细的实际构建块和学习此类表示的标准的需求。我们讨论医学成像和计算机视觉中的应用,强调了在示例钥匙作品中进行的选择。我们通过呈现剩下的挑战和机会来结束。
translated by 谷歌翻译
我们介绍了一个现实的单发网眼的人体头像创作的系统,即简称罗马。使用一张照片,我们的模型估计了特定于人的头部网格和相关的神经纹理,该神经纹理编码局部光度和几何细节。最终的化身是操纵的,可以使用神经网络进行渲染,该神经网络与野外视频数据集上的网格和纹理估计器一起训练。在实验中,我们观察到我们的系统在头部几何恢复和渲染质量方面都具有竞争性的性能,尤其是对于跨人的重新制定。请参阅结果https://samsunglabs.github.io/rome/
translated by 谷歌翻译
通过突出显示为决定贡献最大的输入图像的区域,显着性图已成为使神经网络解释的流行方法。在医学成像中,它们特别适合于在异常定位的背景下解释神经网络。然而,从我们的实验中,它们不太适用于分类问题,其中允许区分不同类别的特征在空间上相关,散射和绝对是非微不足道的。在本文中,我们提出了一种新的范例,以获得更好的可解释性。为此,我们向用户提供相关且易于解释的信息,以便他可以形成自己的意见。我们使用Disentangled的变分自动编码器,潜在表示分为两个组成部分:不可解释的部分和解剖部件。后者占了明确表示不同类别的分类变量。除了提供给定输入样本的类之外,这种模型还通过修改潜在表示中的分类变量的值来改变对另一个类的样本来将样本转换为另一类的样本。这铺平了更容易解释阶级差异的方式。我们说明了这种方法在法医学中髋部骨骼的自动性测定背景下的相关性。模型编码的功能,发现不同类别的功能与专家知识一致。
translated by 谷歌翻译
Intelligent mesh generation (IMG) refers to a technique to generate mesh by machine learning, which is a relatively new and promising research field. Within its short life span, IMG has greatly expanded the generalizability and practicality of mesh generation techniques and brought many breakthroughs and potential possibilities for mesh generation. However, there is a lack of surveys focusing on IMG methods covering recent works. In this paper, we are committed to a systematic and comprehensive survey describing the contemporary IMG landscape. Focusing on 110 preliminary IMG methods, we conducted an in-depth analysis and evaluation from multiple perspectives, including the core technique and application scope of the algorithm, agent learning goals, data types, targeting challenges, advantages and limitations. With the aim of literature collection and classification based on content extraction, we propose three different taxonomies from three views of key technique, output mesh unit element, and applicable input data types. Finally, we highlight some promising future research directions and challenges in IMG. To maximize the convenience of readers, a project page of IMG is provided at \url{https://github.com/xzb030/IMG_Survey}.
translated by 谷歌翻译
由于其语义上的理解和用户友好的可控性,通过三维引导,通过三维引导的面部图像操纵已广泛应用于各种交互式场景。然而,现有的基于3D形式模型的操作方法不可直接适用于域名面,例如非黑色素化绘画,卡通肖像,甚至是动物,主要是由于构建每个模型的强大困难具体面部域。为了克服这一挑战,据我们所知,我们建议使用人为3DMM操纵任意域名的第一种方法。这是通过两个主要步骤实现的:1)从3DMM参数解开映射到潜在的STYLEGO2的潜在空间嵌入,可确保每个语义属性的解除响应和精确的控制; 2)通过实施一致的潜空间嵌入,桥接域差异并使人类3DMM适用于域外面的人类3DMM。实验和比较展示了我们高质量的语义操作方法在各种面部域中的优越性,所有主要3D面部属性可控姿势,表达,形状,反照镜和照明。此外,我们开发了直观的编辑界面,以支持用户友好的控制和即时反馈。我们的项目页面是https://cassiepython.github.io/cddfm3d/index.html
translated by 谷歌翻译
Figure 1: DeepSDF represents signed distance functions (SDFs) of shapes via latent code-conditioned feed-forward decoder networks. Above images are raycast renderings of DeepSDF interpolating between two shapes in the learned shape latent space. Best viewed digitally.
translated by 谷歌翻译
与传统的头像创建管道相反,这是一个昂贵的过程,现代生成方法直接从照片中学习数据分布,而艺术的状态现在可以产生高度的照片现实图像。尽管大量作品试图扩展无条件的生成模型并达到一定程度的可控性,但要确保多视图一致性,尤其是在大型姿势中,仍然具有挑战性。在这项工作中,我们提出了一个3D肖像生成网络,该网络可产生3D一致的肖像,同时根据有关姿势,身份,表达和照明的语义参数可控。生成网络使用神经场景表示在3D中建模肖像,其生成以支持明确控制的参数面模型为指导。尽管可以通过将图像与部分不同的属性进行对比,但可以进一步增强潜在的分离,但在非面积区域(例如,在动画表达式)时,仍然存在明显的不一致。我们通过提出一种体积混合策略来解决此问题,在该策略中,我们通过将动态和静态辐射场融合在一起,形成一个复合输出,并从共同学习的语义场中分割了两个部分。我们的方法在广泛的实验中优于先前的艺术,在自由视点中观看时,在自然照明中产生了逼真的肖像。所提出的方法还证明了真实图像以及室外卡通面孔的概括能力,在实际应用中显示出巨大的希望。其他视频结果和代码将在项目网页上提供。
translated by 谷歌翻译