Learned 3D representations of human faces are useful for computer vision problems such as 3D face tracking and reconstruction from images, as well as graphics applications such as character generation and animation. Traditional models learn a latent representation of a face using linear subspaces or higher-order tensor generalizations. Due to this linearity, they can not capture extreme deformations and nonlinear expressions. To address this, we introduce a versatile model that learns a non-linear representation of a face using spectral convolutions on a mesh surface. We introduce mesh sampling operations that enable a hierarchical mesh representation that captures non-linear variations in shape and expression at multiple scales within the model. In a variational setting, our model samples diverse realistic 3D faces from a multivariate Gaussian distribution. Our training data consists of 20,466 meshes of extreme expressions captured over 12 different subjects. Despite limited training data, our trained model outperforms state-of-the-art face models with 50% lower reconstruction error, while using 75% fewer parameters. We show that, replacing the expression space of an existing state-of-theart face model with our model, achieves a lower reconstruction error. Our data, model and code are available at http://coma.is.tue.mpg.de/.
translated by 谷歌翻译
在本文中,我们展示了Facetunegan,一种新的3D面部模型表示分解和编码面部身份和面部表情。我们提出了对图像到图像翻译网络的第一次适应,该图像已经成功地用于2D域,到3D面几何。利用最近释放的大面扫描数据库,神经网络已经过培训,以便与面部更好的了解,使面部表情转移和中和富有效应面的变异因素。具体而言,我们设计了一种适应基础架构的对抗架构,并使用Spiralnet ++进行卷积和采样操作。使用两个公共数据集(FACESCAPE和COMA),Facetunegan具有比最先进的技术更好的身份分解和面部中和。它还通过预测较近地面真实数据的闪烁形状并且由于源极和目标之间的面部形态过于不同的面部形态而越来越多的不期望的伪像来优异。
translated by 谷歌翻译
The underlying dynamics and patterns of 3D surface meshes deforming over time can be discovered by unsupervised learning, especially autoencoders, which calculate low-dimensional embeddings of the surfaces. To study the deformation patterns of unseen shapes by transfer learning, we want to train an autoencoder that can analyze new surface meshes without training a new network. Here, most state-of-the-art autoencoders cannot handle meshes of different connectivity and therefore have limited to no generalization capacities to new meshes. Also, reconstruction errors strongly increase in comparison to the errors for the training shapes. To address this, we propose a novel spectral CoSMA (Convolutional Semi-Regular Mesh Autoencoder) network. This patch-based approach is combined with a surface-aware training. It reconstructs surfaces not presented during training and generalizes the deformation behavior of the surfaces' patches. The novel approach reconstructs unseen meshes from different datasets in superior quality compared to state-of-the-art autoencoders that have been trained on these shapes. Our transfer learning errors on unseen shapes are 40% lower than those from models learned directly on the data. Furthermore, baseline autoencoders detect deformation patterns of unseen mesh sequences only for the whole shape. In contrast, due to the employed regional patches and stable reconstruction quality, we can localize where on the surfaces these deformation patterns manifest.
translated by 谷歌翻译
在面孔和机构的3D生成模型中学习解除一致,可解释和结构化的潜在代表仍然是一个开放的问题。当需要对身份特征的控制时,问题特别严重。在本文中,我们提出了一种直观但有效的自我监督方法来训练3D形变形自动化器(VAE),鼓励身份特征的解开潜在表示。通过在不同形状上交换任意特征来造成迷你批处理允许定义利用潜在表示中已知差异和相似性的损耗功能。在3D网眼上进行的实验结果表明,最先进的潜在解剖学方法无法解散面部和身体的身份特征。我们所提出的方法适当地解耦了这些特征的产生,同时保持了良好的表示和重建能力。
translated by 谷歌翻译
在这项工作中,我们解决了4D面部表情生成的问题。通常,通过对中性3D面动画来达到表达峰,然后回到中立状态来解决这一问题。但是,在现实世界中,人们表现出更复杂的表情,并从一个表达式转换为另一种表达。因此,我们提出了一个新模型,该模型在不同表达式之间产生过渡,并综合了长长的4D表达式。这涉及三个子问题:(i)建模表达式的时间动力学,(ii)它们之间的学习过渡,以及(iii)变形通用网格。我们建议使用一组3D地标的运动编码表达式的时间演变,我们学会通过训练一个具有歧管值的gan(Motion3dgan)来生成。为了允许生成组成的表达式,该模型接受两个编码起始和结尾表达式的标签。网格的最终顺序是由稀疏的2块网格解码器(S2D-DEC)生成的,该解码器将地标位移映射到已知网格拓扑的密集,每位vertex位移。通过明确处理运动轨迹,该模型完全独立于身份。五个公共数据集的广泛实验表明,我们提出的方法在以前的解决方案方面带来了重大改进,同时保留了良好的概括以看不见数据。
translated by 谷歌翻译
Deep learning has achieved a remarkable performance breakthrough in several fields, most notably in speech recognition, natural language processing, and computer vision. In particular, convolutional neural network (CNN) architectures currently produce state-of-the-art performance on a variety of image analysis tasks such as object detection and recognition. Most of deep learning research has so far focused on dealing with 1D, 2D, or 3D Euclideanstructured data such as acoustic signals, images, or videos. Recently, there has been an increasing interest in geometric deep learning, attempting to generalize deep learning methods to non-Euclidean structured data such as graphs and manifolds, with a variety of applications from the domains of network analysis, computational social science, or computer graphics. In this paper, we propose a unified framework allowing to generalize CNN architectures to non-Euclidean domains (graphs and manifolds) and learn local, stationary, and compositional task-specific features. We show that various non-Euclidean CNN methods previously proposed in the literature can be considered as particular instances of our framework. We test the proposed method on standard tasks from the realms of image-, graphand 3D shape analysis and show that it consistently outperforms previous approaches.
translated by 谷歌翻译
Many scientific fields study data with an underlying structure that is a non-Euclidean space. Some examples include social networks in computational social sciences, sensor networks in communications, functional networks in brain imaging, regulatory networks in genetics, and meshed surfaces in computer graphics. In many applications, such geometric data are large and complex (in the case of social networks, on the scale of billions), and are natural targets for machine learning techniques. In particular, we would like to use deep neural networks, which have recently proven to be powerful tools for a broad range of problems from computer vision, natural language processing, and audio analysis. However, these tools have been most successful on data with an underlying Euclidean or grid-like structure, and in cases where the invariances of these structures are built into networks used to model them.Geometric deep learning is an umbrella term for emerging techniques attempting to generalize (structured) deep neural models to non-Euclidean domains such as graphs and manifolds. The purpose of this paper is to overview different examples of geometric deep learning problems and present available solutions, key difficulties, applications, and future research directions in this nascent field.
translated by 谷歌翻译
3D面部重建是一个具有挑战性的问题,但也是计算机视觉和图形领域的重要任务。最近,许多研究人员对这个问题提请注意,并且已经发表了大量的文章。单个图像重建是3D面部重建的分支之一,在我们的生活中具有大量应用。本文是对从单个图像的3D面部重建最近的文献述评。
translated by 谷歌翻译
In this work, we are interested in generalizing convolutional neural networks (CNNs) from low-dimensional regular grids, where image, video and speech are represented, to high-dimensional irregular domains, such as social networks, brain connectomes or words' embedding, represented by graphs. We present a formulation of CNNs in the context of spectral graph theory, which provides the necessary mathematical background and efficient numerical schemes to design fast localized convolutional filters on graphs. Importantly, the proposed technique offers the same linear computational complexity and constant learning complexity as classical CNNs, while being universal to any graph structure. Experiments on MNIST and 20NEWS demonstrate the ability of this novel deep learning system to learn local, stationary, and compositional features on graphs.
translated by 谷歌翻译
闭塞是无关的面部图像中的常见发生。由于闭塞的存在,这种面部图像的单个图像3D重建经常受到腐败。此外,虽然在遮挡区域中具有多个3D重建,但是现有方法仅限于仅产生单个解决方案。为了解决这两种挑战,我们呈现了不同的3DFace,专门设计用于从单个遮挡的面部图像同时产生一系列多样化的3D重建集。它由三个组成部分组成:全局+局部形状拟合过程,基于图形神经网络的网格VAE,以及促进迭代优化过程的决定性点过程的多样性。在闭塞面上的3D重建的定量和定性比较表明,多样化3dface可以估计与目标图像中的可见区域一致的3D形状,同时在遮挡区域上表现出高而逼真的分集。在面部图像上由掩模,眼镜和其他随机物体封闭,不同的3dface在与基线相比,在遮挡区域上产生3D形状的3D形状的分布。此外,我们最接近地面真理的样品比现有方法的单数重建降低了40%。
translated by 谷歌翻译
3D漫画是对人脸的夸张的3D描述。本文的目的是对紧凑的参数空间中的3D漫画的变化进行建模,以便我们可以为处理3D漫画变形提供有用的数据驱动工具包。为了实现目标,我们提出了一个基于MLP的框架,用于构建可变形的表面模型,该模型采用潜在代码并产生3D表面。在框架中,警笛MLP模拟了在固定模板表面上采用3D位置并返回输入位置的3D位移向量的函数。我们通过学习采用潜在代码并产生MLP参数的超网络来创建3D表面的变化。一旦了解到,我们的可变形模型为3D漫画提供了一个不错的编辑空间,支持基于标签的语义编辑和基于尖的基于尖的变形,这两者都产生了高度夸张和自然的3D讽刺形状。我们还展示了可变形模型的其他应用,例如自动3D漫画创建。
translated by 谷歌翻译
基于简单的扩散层对空间通信非常有效的洞察力,我们对3D表面进行深度学习的新的通用方法。由此产生的网络是自动稳健的,以改变表面的分辨率和样品 - 一种对实际应用至关重要的基本属性。我们的网络可以在各种几何表示上离散化,例如三角网格或点云,甚至可以在一个表示上培训然后应用于另一个表示。我们优化扩散的空间支持,作为连续网络参数,从纯粹的本地到完全全球范围,从而消除手动选择邻域大小的负担。该方法中唯一的其他成分是在每个点处独立地施加的多层的Perceptron,以及用于支持方向滤波器的空间梯度特征。由此产生的网络简单,坚固,高效。这里,我们主要专注于三角网格表面,并且展示了各种任务的最先进的结果,包括表面分类,分割和非刚性对应。
translated by 谷歌翻译
We propose a novel 3D morphable model for complete human heads based on hybrid neural fields. At the core of our model lies a neural parametric representation which disentangles identity and expressions in disjoint latent spaces. To this end, we capture a person's identity in a canonical space as a signed distance field (SDF), and model facial expressions with a neural deformation field. In addition, our representation achieves high-fidelity local detail by introducing an ensemble of local fields centered around facial anchor points. To facilitate generalization, we train our model on a newly-captured dataset of over 2200 head scans from 124 different identities using a custom high-end 3D scanning setup. Our dataset significantly exceeds comparable existing datasets, both with respect to quality and completeness of geometry, averaging around 3.5M mesh faces per scan. Finally, we demonstrate that our approach outperforms state-of-the-art methods by a significant margin in terms of fitting error and reconstruction quality.
translated by 谷歌翻译
因果推理提供了一种语言,以提出纯粹统计关联以外的重要介入和反事实问题。例如,在医学成像中,我们可能希望研究遗传,环境或生活方式因素对解剖表型正常和病理变异的因果关系。但是,尽管可以可靠地构建从自动图像分割中提取的3D表面网格的解剖形状模型,但缺乏计算工具来实现有关形态变化的因果推理。为了解决这个问题,我们提出了深层结构性因果形状模型(CSM),该模型利用了高质量的网格生成技术,从几何深度学习,在深层结构性因果模型的表达框架内。 CSM可以通过反事实网格产生来实现特定于受试者的预后(“如果患者大十岁,该患者的大脑结构将如何变化?”),这与大多数当前有关纯粹人口级统计形状建模的作品形成鲜明对比。我们通过许多定性和定量实验利用了3D脑结构的大数据集,证明了Pearl因果关系层次结构的所有级别CSM的能力。
translated by 谷歌翻译
形状空间学习的任务涉及使用良好的概括性属性映射到从潜在表示空间的列车组。通常,真实世界的形状系列具有对称性,可以定义为不改变形状本质的转换。在形状空间学习中纳入对称性的自然方式是要求将其映射到形状空间(编码器)和从形状空间(解码器)映射到相关的对称。在本文中,我们通过引入两个贡献,提出了一种在编码器和解码器中融入设备和解码器的框架:(i)适应建设通用,高效和最大富有表现力的Autorencoders的最近帧平均(FA)框架; (ii)构建自动化器等于分段欧几里德运动的分段应用于形状的不同部分。据我们所知,这是第一个完全分段的欧几里德的欧洲等自动化器建设。培训我们的框架很简单:它使用标准的重建损失,不需要引入新的损失。我们的体系结构由标准(骨干网)架构构成,具有适当的帧平均,使其成为等效。使用隐式的神经表示,在两个刚性形状数据集上测试我们的框架,并使用基于网格的神经网络的铰接形状数据集显示出技术的概括,以通过大边缘改善相关基线。特别地,我们的方法表明了概括铰接姿势的概括性的显着改善。
translated by 谷歌翻译
在过去几年中,许多面部分析任务已经完成了惊人的性能,其中应用包括来自单个“野外”图像的面部生成和3D面重建。尽管如此,据我们所知,没有方法可以从“野外”图像中产生渲染的高分辨率3D面,并且这可以归因于:(a)可用数据的跨度进行培训(b)缺乏可以成功应用于非常高分辨率数据的强大方法。在这项工作中,我们介绍了一种能够从单个“野外”图像中重建光电型渲染3D面部几何和BRDF的第一种方法。我们捕获了一个大型的面部形状和反射率,我们已经公开了。我们用精确的面部皮肤漫射和镜面反射,自遮挡和地下散射近似来定义快速面部光电型拟型渲染方法。有了这一点,我们训练一个网络,将面部漫射和镜面BRDF组件与烘焙照明的形状和质地一起脱颖而出,以最先进的3DMM配件方法重建。我们的方法通过显着的余量优于现有技术,并从单个低分辨率图像重建高分辨率3D面,这可以在各种应用中呈现,并桥接不一体谷。
translated by 谷歌翻译
我们提出了一个框架来学习一个结构化的潜在空间来代表4D人体运动,其中每个潜在向量都编码整个3D人类形状的全部运动。一方面,存在一些数据驱动的骨骼动画模型,提出了时间密集运动信号的运动空间,但基于几何稀疏的运动学表示。另一方面,存在许多方法来构建密集的3D几何形状的形状空间,但对于静态帧。我们将两个概念汇总在一起,提出一个运动空间,该运动空间在时间和几何上都很密集。经过训练后,我们的模型将基于低维潜在空间中的单个点生成多帧序列。该潜在空间是构建为结构化的,因此类似的运动形成簇。它还嵌入了潜在矢量中的持续时间变化,允许语义上的接近序列,这些序列仅因时间展开而不同以共享相似的潜在矢量。我们通过实验证明了潜在空间的结构特性,并表明它可用于在不同动作之间生成合理的插值。我们还将模型应用于4D人类运动的完成,显示其有希望学习人类运动时空特征的能力。
translated by 谷歌翻译
近年来,人类面孔的影子化化身已经走了很长一段路,但是该地区的研究受到缺乏公开可用的高质量数据集的限制。在这项工作中,我们介绍了Multiface,这是一种新的多视图,高分辨率的人脸数据集,该数据集是从13个身份的神经面部渲染研究中收集的13个身份。我们介绍了Mugsy,这是一种大型多摄像机设备,可捕获面部表现的高分辨率同步视频。 Multiface的目的是缩小学术界高质量数据的可访问性的差距,并使VR触觉研究能够进行研究。随着数据集的释放,我们对不同模型体系结构对模型的新观点和表达式的插值能力进行消融研究。通过有条件的VAE模型作为我们的基线,我们发现添加空间偏见,纹理翘曲场和残差连接可改善新型视图合成的性能。我们的代码和数据可在以下网址获得:https://github.com/facebookresearch/multiface
translated by 谷歌翻译
我们提出了神经头头像,这是一种新型神经表示,其明确地模拟了可动画的人体化身的表面几何形状和外观,可用于在依赖数字人类的电影或游戏行业中的AR / VR或其他应用中的电话会议。我们的代表可以从单眼RGB肖像视频中学到,该视频具有一系列不同的表达和视图。具体地,我们提出了一种混合表示,其由面部的粗糙形状和表达式和两个前馈网络组成的混合表示,以及预测底层网格的顶点偏移以及视图和表达依赖性纹理。我们证明,该表示能够准确地外推到看不见的姿势和观点,并在提供尖锐的纹理细节的同时产生自然表达。与先前的磁头头像上的作品相比,我们的方法提供了与标准图形管道兼容的完整人体头(包括头发)的分解形状和外观模型。此外,就重建质量和新型观看合成而定量和定性地优于现有技术的当前状态。
translated by 谷歌翻译
To facilitate the analysis of human actions, interactions and emotions, we compute a 3D model of human body pose, hand pose, and facial expression from a single monocular image. To achieve this, we use thousands of 3D scans to train a new, unified, 3D model of the human body, SMPL-X, that extends SMPL with fully articulated hands and an expressive face. Learning to regress the parameters of SMPL-X directly from images is challenging without paired images and 3D ground truth. Consequently, we follow the approach of SMPLify, which estimates 2D features and then optimizes model parameters to fit the features. We improve on SMPLify in several significant ways: (1) we detect 2D features corresponding to the face, hands, and feet and fit the full SMPL-X model to these; (2) we train a new neural network pose prior using a large MoCap dataset; (3) we define a new interpenetration penalty that is both fast and accurate; (4) we automatically detect gender and the appropriate body models (male, female, or neutral); (5) our PyTorch implementation achieves a speedup of more than 8× over Chumpy. We use the new method, SMPLify-X, to fit SMPL-X to both controlled images and images in the wild. We evaluate 3D accuracy on a new curated dataset comprising 100 images with pseudo ground-truth. This is a step towards automatic expressive human capture from monocular RGB data. The models, code, and data are available for research purposes at https://smpl-x.is.tue.mpg.de.
translated by 谷歌翻译