To facilitate the analysis of human actions, interactions and emotions, we compute a 3D model of human body pose, hand pose, and facial expression from a single monocular image. To achieve this, we use thousands of 3D scans to train a new, unified, 3D model of the human body, SMPL-X, that extends SMPL with fully articulated hands and an expressive face. Learning to regress the parameters of SMPL-X directly from images is challenging without paired images and 3D ground truth. Consequently, we follow the approach of SMPLify, which estimates 2D features and then optimizes model parameters to fit the features. We improve on SMPLify in several significant ways: (1) we detect 2D features corresponding to the face, hands, and feet and fit the full SMPL-X model to these; (2) we train a new neural network pose prior using a large MoCap dataset; (3) we define a new interpenetration penalty that is both fast and accurate; (4) we automatically detect gender and the appropriate body models (male, female, or neutral); (5) our PyTorch implementation achieves a speedup of more than 8× over Chumpy. We use the new method, SMPLify-X, to fit SMPL-X to both controlled images and images in the wild. We evaluate 3D accuracy on a new curated dataset comprising 100 images with pseudo ground-truth. This is a step towards automatic expressive human capture from monocular RGB data. The models, code, and data are available for research purposes at https://smpl-x.is.tue.mpg.de.
translated by 谷歌翻译
We describe the first method to automatically estimate the 3D pose of the human body as well as its 3D shape from a single unconstrained image. We estimate a full 3D mesh and show that 2D joints alone carry a surprising amount of information about body shape. The problem is challenging because of the complexity of the human body, articulation, occlusion, clothing, lighting, and the inherent ambiguity in inferring 3D from 2D. To solve this, we first use a recently published CNN-based method, DeepCut, to predict (bottom-up) the 2D body joint locations. We then fit (top-down) a recently published statistical body shape model, called SMPL, to the 2D joints. We do so by minimizing an objective function that penalizes the error between the projected 3D model joints and detected 2D joints. Because SMPL captures correlations in human shape across the population, we are able to robustly fit it to very little data. We further leverage the 3D model to prevent solutions that cause interpenetration. We evaluate our method, SMPLify, on the Leeds Sports, HumanEva, and Human3.6M datasets, showing superior pose accuracy with respect to the state of the art.
translated by 谷歌翻译
人类将他们的手和身体一起移动,沟通和解决任务。捕获和复制此类协调活动对于虚拟字符至关重要,以实际行为行为。令人惊讶的是,大多数方法分别对待身体和手的3D建模和跟踪。在这里,我们制定了一种手和身体的型号,并将其与全身4D序列合理。当扫描或捕获3D中的全身时,手很小,通常是部分闭塞,使其形状和难以恢复。为了应对低分辨率,闭塞和噪音,我们开发了一种名为Mano(具有铰接和非刚性变形的手模型)的新型号。曼诺从大约1000个高分辨率的3D扫描中学到了31个受试者的手中的大约一定的手。该模型是逼真的,低维,捕获非刚性形状的姿势变化,与标准图形封装兼容,可以适合任何人类的手。 Mano提供从手姿势的紧凑型映射,以构成混合形状校正和姿势协同效应的线性歧管。我们将Mano附加到标准参数化3D体形状模型(SMPL),导致完全铰接的身体和手部模型(SMPL + H)。我们通过用4D扫描仪捕获的综合体,自然,自然,自然的受试者的活动来说明SMPL + H.该配件完全自动,并导致全身型号,自然地移动详细的手动运动和在全身性能捕获之前未见的现实主义。模型和数据在我们的网站上自由用于研究目的(http://mano.is.tue.mpg.de)。
translated by 谷歌翻译
Figure 1: Frankenstein (silver) and Adam (gold). This paper presents a 3D human model capable of concurrently tracking the large-scale posture of the body along with the smaller details of a persons facial expressions and hand gestures.
translated by 谷歌翻译
推断人类场景接触(HSC)是了解人类如何与周围环境相互作用的第一步。尽管检测2D人类对象的相互作用(HOI)和重建3D人姿势和形状(HPS)已经取得了重大进展,但单个图像的3D人习惯接触的推理仍然具有挑战性。现有的HSC检测方法仅考虑几种类型的预定义接触,通常将身体和场景降低到少数原语,甚至忽略了图像证据。为了预测单个图像的人类场景接触,我们从数据和算法的角度解决了上述局限性。我们捕获了一个名为“真实场景,互动,联系和人类”的新数据集。 Rich在4K分辨率上包含多视图室外/室内视频序列,使用无标记运动捕获,3D身体扫描和高分辨率3D场景扫描捕获的地面3D人体。 Rich的一个关键特征是它还包含身体上精确的顶点级接触标签。使用Rich,我们训练一个网络,该网络可预测单个RGB图像的密集车身场景接触。我们的主要见解是,接触中的区域总是被阻塞,因此网络需要能够探索整个图像以获取证据。我们使用变压器学习这种非本地关系,并提出新的身体场景接触变压器(BSTRO)。很少有方法探索3D接触;那些只专注于脚的人,将脚接触作为后处理步骤,或从身体姿势中推断出无需看现场的接触。据我们所知,BSTRO是直接从单个图像中直接估计3D身体场景接触的方法。我们证明,BSTRO的表现明显优于先前的艺术。代码和数据集可在https://rich.is.tue.mpg.de上获得。
translated by 谷歌翻译
Input Reconstruction Side and top down view Part Segmentation Input Reconstruction Side and top down view Part Segmentation Figure 1: Human Mesh Recovery (HMR): End-to-end adversarial learning of human pose and shape. We describe a real time framework for recovering the 3D joint angles and shape of the body from a single RGB image. The first two rowsshow results from our model trained with some 2D-to-3D supervision, the bottom row shows results from a model that is trained in a fully weakly-supervised manner without using any paired 2D-to-3D supervision. We infer the full 3D body even in case of occlusions and truncations. Note that we capture head and limb orientations.
translated by 谷歌翻译
我们提出了一种基于优化的新型范式,用于在图像和扫描上拟合3D人类模型。与直接回归输入图像中低维统计体模型(例如SMPL)的参数的现有方法相反,我们训练了每个vertex神经场网络的集合。该网络以分布式的方式预测基于当前顶点投影处提取的神经特征的顶点下降方向。在推断时,我们在梯度降低的优化管道中采用该网络,称为LVD,直到其收敛性为止,即使将所有顶点初始化为单个点,通常也会以一秒钟的分数出现。一项详尽的评估表明,我们的方法能够捕获具有截然不同的身体形状的穿着的人体,与最先进的人相比取得了重大改进。 LVD也适用于人类和手的3D模型配合,为此,我们以更简单,更快的方法对SOTA显示出显着改善。
translated by 谷歌翻译
虽然从图像中回归3D人类的方法迅速发展,但估计的身体形状通常不会捕获真正的人形状。这是有问题的,因为对于许多应用,准确的身体形状与姿势一样重要。身体形状准确性差姿势准确性的关键原因是缺乏数据。尽管人类可以标记2D关节,并且这些约束3D姿势,但“标记” 3D身体形状并不容易。由于配对的数据与图像和3D身体形状很少见,因此我们利用了两个信息来源:(1)我们收集了各种“时尚”模型的互联网图像,以及一系列的人体测量值; (2)我们为3D身体网眼和模型图像收集语言形状属性。综上所述,这些数据集提供了足够的约束来推断密集的3D形状。我们利用几种新型方法来利用人体测量和语言形状属性来训练称为Shapy的神经网络,从而从RGB图像中回归了3D人类的姿势和形状。我们在公共基准测试上评估shapy,但请注意,它们要么缺乏明显的身体形状变化,地面真实形状或衣服变化。因此,我们收集了一个新的数据集,用于评估3D人类形状估计,称为HBW,其中包含“野生人体”的照片,我们为其具有地面3D身体扫描。在这个新的基准测试中,Shapy在3D身体估计的任务上的最先进方法极大地胜过。这是第一次演示,即可以从易于观察的人体测量和语言形状属性中训练来自图像的3D体形回归。我们的模型和数据可在以下网址获得:shapy.is.tue.mpg.de
translated by 谷歌翻译
了解来自第一人称观点的社交互动对于许多应用来说至关重要,从辅助机器人到AR / VR。谈论相互作用的第一步是理解人类的姿势和形状。但是,该领域的研究目前受到数据缺乏的阻碍。现有数据集根据大小,注释,地面真实捕获方式或相互作用的多样性有限。我们通过提出EGOBODY来解决这一缺点,这是一个用于复杂3D场景中的社交交互的新型大规模数据集。我们采用Microsoft Hololens2耳机来记录富裕的EGEntric数据流(包括RGB,深度,眼睛凝视,头部和手动跟踪)。为了获得准确的3D地面真理,我们将耳机用多kinect钻机校准并配合富有呈现的SMPL-X体网格到多视图RGB-D帧,重建3D人类姿势和相对于场景的形状。我们收集68个序列,跨越不同的社会学互动类别,并提出了从自我监视视图的3D全体姿态和形状估计的第一个基准。我们的数据集和代码将在https://sanweiliti.github.io/egobody/egobody.html中进行研究。
translated by 谷歌翻译
This work addresses the problem of estimating the full body 3D human pose and shape from a single color image. This is a task where iterative optimization-based solutions have typically prevailed, while Convolutional Networks (ConvNets) have suffered because of the lack of training data and their low resolution 3D predictions. Our work aims to bridge this gap and proposes an efficient and effective direct prediction method based on ConvNets. Central part to our approach is the incorporation of a parametric statistical body shape model (SMPL) within our end-to-end framework. This allows us to get very detailed 3D mesh results, while requiring estimation only of a small number of parameters, making it friendly for direct network prediction. Interestingly, we demonstrate that these parameters can be predicted reliably only from 2D keypoints and masks. These are typical outputs of generic 2D human analysis ConvNets, allowing us to relax the massive requirement that images with 3D shape ground truth are available for training. Simultaneously, by maintaining differentiability, at training time we generate the 3D mesh from the estimated parameters and optimize explicitly for the surface using a 3D per-vertex loss. Finally, a differentiable renderer is employed to project the 3D mesh to the image, which enables further refinement of the network, by optimizing for the consistency of the projection with 2D annotations (i.e., 2D keypoints or masks). The proposed approach outperforms previous baselines on this task and offers an attractive solution for direct prediction of 3D shape from a single color image.
translated by 谷歌翻译
人类不断与日常对象互动以完成任务。为了了解这种相互作用,计算机需要从观察全身与场景的全身相互作用的相机中重建这些相互作用。由于身体和物体之间的阻塞,运动模糊,深度/比例模棱两可以及手和可抓握的物体零件的低图像分辨率,这是具有挑战性的。为了使问题可以解决,社区要么专注于互动的手,忽略身体或互动的身体,无视双手。 Grab数据集解决了灵活的全身互动,但使用基于标记的MOCAP并缺少图像,而行为则捕获了身体对象互动的视频,但缺乏手动细节。我们使用参数全身模型SMPL-X和已知的对象网格来解决一种新的方法,该方法与Intercap的先前工作局限性,该方法是一种新的方法,可重建从多视图RGB-D数据进行交互的整体和对象。为了应对上述挑战,Intercap使用了两个关键观察:(i)可以使用手和物体之间的接触来改善两者的姿势估计。 (ii)Azure Kinect传感器使我们能够建立一个简单的多视图RGB-D捕获系统,该系统在提供合理的相机间同步时最小化遮挡的效果。使用此方法,我们捕获了Intercap数据集,其中包含10个受试者(5名男性和5个女性)与10个各种尺寸和负担的物体相互作用,包括与手或脚接触。 Intercap总共有223个RGB-D视频,产生了67,357个多视图帧,每个帧包含6个RGB-D图像。我们的方法为每个视频框架提供了伪真正的身体网格和对象。我们的Intercap方法和数据集填补了文献中的重要空白,并支持许多研究方向。我们的数据和代码可用于研究目的。
translated by 谷歌翻译
目前用于学习现实和可动画3D穿衣服的方法需要带有仔细控制的用户的构成3D扫描或2D图像。相比之下,我们的目标是从不受约束的姿势中只有2D人的人们学习化身。给定一组图像,我们的方法估计来自每个图像的详细3D表面,然后将它们组合成一个可动画的化身。隐式功能非常适合第一个任务,因为他们可以捕获像头发或衣服等细节。然而,目前的方法对各种人类的姿势并不稳健,并且通常会产生破碎或肢体的3D表面,缺少细节或非人形状。问题是这些方法使用对全局姿势敏感的全局特征编码器。为了解决这个问题,我们提出图标(“从正规中获得的隐式衣物人类”),它使用本地特征。图标有两个主要模块,两者都利用SMPL(-X)正文模型。首先,图标Infers详细的衣服 - 人类法线(前/后)在SMPL(-X)法线上。其次,可视性感知隐式表面回归系统产生人占用场的ISO表面。重要的是,在推断时间下,反馈回路在使用推断的布料正线改进SMPL(-X)网格之间交替,然后改装正常。给定多种姿势的多个重建帧,我们使用扫描来从中生成可动画的化身。对Agora和Cape数据集的评估显示,即使具有大量有限的培训数据,图标越优于重建中的最新状态。另外,它对分布外样品进行更强大,例如,野外的姿势/图像和帧外裁剪。图标从野外图像中迈向强大的3D穿上人体重建。这使得能够使用个性化和天然姿势依赖布变形来直接从视频创建化身。
translated by 谷歌翻译
Recent work has shown the benefits of synthetic data for use in computer vision, with applications ranging from autonomous driving to face landmark detection and reconstruction. There are a number of benefits of using synthetic data from privacy preservation and bias elimination to quality and feasibility of annotation. Generating human-centered synthetic data is a particular challenge in terms of realism and domain-gap, though recent work has shown that effective machine learning models can be trained using synthetic face data alone. We show that this can be extended to include the full body by building on the pipeline of Wood et al. to generate synthetic images of humans in their entirety, with ground-truth annotations for computer vision applications. In this report we describe how we construct a parametric model of the face and body, including articulated hands; our rendering pipeline to generate realistic images of humans based on this body model; an approach for training DNNs to regress a dense set of landmarks covering the entire body; and a method for fitting our body model to dense landmarks predicted from multiple views.
translated by 谷歌翻译
大多数先前的作品在从图像中感知3D人类的作品是孤立的,而没有周围的环境。但是,人类一直在与周围的物体互动,因此呼吁不仅可以推理人类,而且可以推理对象及其相互作用的模型。由于人类与物体之间的严重阻塞,不同的相互作用类型和深度歧义,问题极具挑战性。在本文中,我们介绍了一种新颖的方法,该方法学会了从单个RGB图像中共同重建人和物体。乔尔从最近的隐性表面学习和基于经典模型的拟合方面的进步中汲取灵感。我们计算人类和对象的神经重建,该神经用两个无符号距离字段隐式表示,一个对应物的对应字段和一个对象姿势场。这使我们能够在相互作用的推理的同时,可牢固地拟合参数的身体模型和3D对象模板。此外,先前的像素对齐的隐式学习方法使用合成数据并做出实际数据中未满足的假设。我们提出了一个优雅的深度缩放,可以在真实数据上进行更有效的形状学习。实验表明,我们的联合重建通过提出的策略学到了明显优于SOTA。我们的代码和型号可在https://virtualhumans.mpi-inf.mpg.de/chore上找到
translated by 谷歌翻译
以准确的,稳健和快速的方式拟合人体,手或面对稀疏输入信号的参数模型,这具有重要的是在AR和VR场景中显着改善浸入。解决这些问题的系统中的一个常见的第一步是直接从输入数据重新分配参数模型的参数。这种方法是快速,稳健的,并且是迭代最小化算法的良好起点。后者搜索最小的能量函数,通常由编码关于问题的结构的知识的数据项和前沿组成。虽然这无疑是一个非常成功的食谱,但前锋往往是手工定义的启发式,发现不同术语之间的正确平衡,以实现高质量的结果是一个非琐碎的任务。此外,转换和优化这些系统以表现方式运行,需要定制实现,要求从工程师和域专家进行大量时间投资。在这项工作中,我们建立了近期学习优化的进步,并提出了由Classic Levenberg-Marquardt算法启发的更新规则。我们展示了所提出的神经优化器对从2D地标的头戴式装置和面部配件的3D体表估计问题的有效性。我们的方法可以很容易地应用于新的模型拟合问题,并提供竞争替代方案,在准确性和速度方面都提供了良好的调谐“传统”模型拟合管道。
translated by 谷歌翻译
新兴的元应用需要人类手的可靠,准确和逼真的复制品,以便在物理世界中进行复杂的操作。虽然真实的人手代表了骨骼,肌肉,肌腱和皮肤之间最复杂的协调之一,但最先进的技术一致专注于仅建模手的骨架。在本文中,我们提出了Nimble,这是一种新型的参数手模型,其中包括缺少的密钥组件,将3D手模型带入了新的现实主义水平。我们首先在最近的磁共振成像手(MRI手)数据集上注释肌肉,骨骼和皮肤,然后在数据集中的单个姿势和受试者上注册一个体积模板手。敏捷由20个骨头组成,作为三角形网格,7个肌肉群作为四面体网眼和一个皮肤网。通过迭代形状的注册和参数学习,它进一步产生形状的混合形状,姿势混合形状和关节回归器。我们证明将敏捷性应用于建模,渲染和视觉推理任务。通过强制执行内部骨骼和肌肉以符合解剖学和运动学规则,Nimble可以使3D手动画为前所未有的现实主义。为了建模皮肤的外观,我们进一步构建了一个光度法,以获取高质量的纹理和正常地图,以模型皱纹和棕榈印刷。最后,敏捷还通过合成丰富的数据或直接作为推理网络中的可区分层来使基于学习的手姿势和形状估计受益。
translated by 谷歌翻译
从单眼RGB图像中捕获的3D人类运动捕获符合受试者与复杂且可能可变形的环境的相互作用的相互作用是一个非常具有挑战性,不足和探索不足的问题。现有方法仅薄弱地解决它,并且当人类与场景表面互动时,通常不会建模可能发生的表面变形。相比之下,本文提出了mocapdeform,即单眼3D人体运动捕获的新框架,该框架是第一个明确模拟3D场景的非刚性变形,以改善3D人体姿势估计和可变形环境的重建。 Mocapdeform接受单眼RGB视频,并在相机空间中对齐一个3D场景。它首先使用基于新的射线广播的策略将输入单眼视频中的主题以及密集的触点标签进行定位。接下来,我们的人类环境相互作用约束被利用以共同优化全局3D人类姿势和非刚性表面变形。 Mocapdeform比在几个数据集上的竞争方法获得了更高的精度,包括我们新记录的具有变形背景场景的方法。
translated by 谷歌翻译
The combination of artist-curated scans, and deep implicit functions (IF), is enabling the creation of detailed, clothed, 3D humans from images. However, existing methods are far from perfect. IF-based methods recover free-form geometry but produce disembodied limbs or degenerate shapes for unseen poses or clothes. To increase robustness for these cases, existing work uses an explicit parametric body model to constrain surface reconstruction, but this limits the recovery of free-form surfaces such as loose clothing that deviates from the body. What we want is a method that combines the best properties of implicit and explicit methods. To this end, we make two key observations: (1) current networks are better at inferring detailed 2D maps than full-3D surfaces, and (2) a parametric model can be seen as a "canvas" for stitching together detailed surface patches. ECON infers high-fidelity 3D humans even in loose clothes and challenging poses, while having realistic faces and fingers. This goes beyond previous methods. Quantitative, evaluation of the CAPE and Renderpeople datasets shows that ECON is more accurate than the state of the art. Perceptual studies also show that ECON's perceived realism is better by a large margin. Code and models are available for research purposes at https://xiuyuliang.cn/econ
translated by 谷歌翻译
Model-based human pose estimation is currently approached through two different paradigms. Optimizationbased methods fit a parametric body model to 2D observations in an iterative manner, leading to accurate imagemodel alignments, but are often slow and sensitive to the initialization. In contrast, regression-based methods, that use a deep network to directly estimate the model parameters from pixels, tend to provide reasonable, but not pixel accurate, results while requiring huge amounts of supervision. In this work, instead of investigating which approach is better, our key insight is that the two paradigms can form a strong collaboration. A reasonable, directly regressed estimate from the network can initialize the iterative optimization making the fitting faster and more accurate. Similarly, a pixel accurate fit from iterative optimization can act as strong supervision for the network. This is the core of our proposed approach SPIN (SMPL oPtimization IN the loop). The deep network initializes an iterative optimization routine that fits the body model to 2D joints within the training loop, and the fitted estimate is subsequently used to supervise the network. Our approach is self-improving by nature, since better network estimates can lead the optimization to better solutions, while more accurate optimization fits provide better supervision for the network. We demonstrate the effectiveness of our approach in different settings, where 3D ground truth is scarce, or not available, and we consistently outperform the state-of-the-art model-based pose estimation approaches by significant margins. The project website with videos, results, and code can be found at https://seas.upenn.edu/ ˜nkolot/projects/spin.
translated by 谷歌翻译
and ACCAD [5] datasets. The input is sparse markers and the output is SMPL body models.
translated by 谷歌翻译