在本文中，提出了一种称为POP-Net的实时方法，以预测来自深度图像的多人3D。 POP-Net了解到在一次拍摄中预测自下而上的部分表示和自上而下的全球姿势。具体地，介绍了一种名为截断的零件位移场（TPDF）的新的零级表示，这使得明确的融合过程能够统一自下而上部分检测和全局姿势检测的优点。同时，引入了有效的模式选择方案以自动解决全局姿势和部分检测之间的冲突案例。最后，由于缺乏用于开发多人3D姿势估计的高质量深度数据集，我们将多人3D人类姿势数据集（MP-3DHP）引入新的基准。 MP-3DHP旨在在模型培训中实现有效的多人和背景数据增强，并在不受控制的多人场景下评估3D人类姿势估计。我们表明Pop-Net在MP-3DHP和广泛使用的ITOP数据集上实现了最先进的结果，并对多人处理的效率具有显着的优势。为了证明我们的算法管道的应用之一，我们还显示了由我们计算的3D关节位置驱动的虚拟化身的结果。 MP-3DHP数据集和评估代码已提供：https://github.com/oppo-us-research/pop-net。

我们提出了一种称为DPODV2（密集姿势对象检测器）的三个阶段6 DOF对象检测方法，该方法依赖于致密的对应关系。我们将2D对象检测器与密集的对应关系网络和多视图姿势细化方法相结合，以估计完整的6 DOF姿势。与通常仅限于单眼RGB图像的其他深度学习方法不同，我们提出了一个统一的深度学习网络，允许使用不同的成像方式（RGB或DEPTH）。此外，我们提出了一种基于可区分渲染的新型姿势改进方法。主要概念是在多个视图中比较预测并渲染对应关系，以获得与所有视图中预测的对应关系一致的姿势。我们提出的方法对受控设置中的不同数据方式和培训数据类型进行了严格的评估。主要结论是，RGB在对应性估计中表现出色，而如果有良好的3D-3D对应关系，则深度有助于姿势精度。自然，他们的组合可以实现总体最佳性能。我们进行广泛的评估和消融研究，以分析和验证几个具有挑战性的数据集的结果。 DPODV2在所有这些方面都取得了出色的成果，同时仍然保持快速和可扩展性，独立于使用的数据模式和培训数据的类型

Estimating 6D poses of objects from images is an important problem in various applications such as robot manipulation and virtual reality. While direct regression of images to object poses has limited accuracy, matching rendered images of an object against the input image can produce accurate results. In this work, we propose a novel deep neural network for 6D pose matching named DeepIM. Given an initial pose estimation, our network is able to iteratively refine the pose by matching the rendered image against the observed image. The network is trained to predict a relative pose transformation using a disentangled representation of 3D location and 3D orientation and an iterative training process. Experiments on two commonly used benchmarks for 6D pose estimation demonstrate that DeepIM achieves large improvements over stateof-the-art methods. We furthermore show that DeepIM is able to match previously unseen objects.

尽管基于体素的方法已经获得了来自多摄像头的多人3D姿势估计的有希望的结果，但它们具有沉重的计算负担，尤其是对于大型场景。我们提出了更快的素素，以通过将特征体积重新投影到三个二维坐标平面并分别估算x，y，z坐标来解决挑战。为此，我们首先通过分别基于投影到XY平面和Z轴的体积功能来估算2D框及其高度，首先通过一个3D边界框来定位每个人。然后，对于每个人，我们分别估算三个坐标平面的部分关节坐标，然后将其融合以获得最终的3D姿势。该方法不含昂贵的3D-CNN，并将其素的速度提高了十倍，同时作为最先进的方法的竞争精度，证明了其在实时应用中的潜力。

We introduce an approach for recovering the 6D pose of multiple known objects in a scene captured by a set of input images with unknown camera viewpoints. First, we present a single-view single-object 6D pose estimation method, which we use to generate 6D object pose hypotheses. Second, we develop a robust method for matching individual 6D object pose hypotheses across different input images in order to jointly estimate camera viewpoints and 6D poses of all objects in a single consistent scene. Our approach explicitly handles object symmetries, does not require depth measurements, is robust to missing or incorrect object hypotheses, and automatically recovers the number of objects in the scene. Third, we develop a method for global scene refinement given multiple object hypotheses and their correspondences across views. This is achieved by solving an object-level bundle adjustment problem that refines the poses of cameras and objects to minimize the reprojection error in all views. We demonstrate that the proposed method, dubbed Cosy-Pose, outperforms current state-of-the-art results for single-view and multi-view 6D object pose estimation by a large margin on two challenging benchmarks: the YCB-Video and T-LESS datasets. Code and pre-trained models are available on the project webpage. 5

Accurate whole-body multi-person pose estimation and tracking is an important yet challenging topic in computer vision. To capture the subtle actions of humans for complex behavior analysis, whole-body pose estimation including the face, body, hand and foot is essential over conventional body-only pose estimation. In this paper, we present AlphaPose, a system that can perform accurate whole-body pose estimation and tracking jointly while running in realtime. To this end, we propose several new techniques: Symmetric Integral Keypoint Regression (SIKR) for fast and fine localization, Parametric Pose Non-Maximum-Suppression (P-NMS) for eliminating redundant human detections and Pose Aware Identity Embedding for jointly pose estimation and tracking. During training, we resort to Part-Guided Proposal Generator (PGPG) and multi-domain knowledge distillation to further improve the accuracy. Our method is able to localize whole-body keypoints accurately and tracks humans simultaneously given inaccurate bounding boxes and redundant detections. We show a significant improvement over current state-of-the-art methods in both speed and accuracy on COCO-wholebody, COCO, PoseTrack, and our proposed Halpe-FullBody pose estimation dataset. Our model, source codes and dataset are made publicly available at https://github.com/MVIG-SJTU/AlphaPose.

人际关系的阻塞和深度歧义使估计单眼多人的3D姿势是以摄像头为中心的坐标，这是一个具有挑战性的问题。典型的自上而下框架具有高计算冗余，并具有额外的检测阶段。相比之下，自下而上的方法的计算成本较低，因为它们受人数的影响较小。但是，大多数现有的自下而上方法将以摄像头3D为中心的人姿势估计视为两个无关的子任务：2.5D姿势估计和以相机为中心的深度估计。在本文中，我们提出了一个统一模型，该模型利用这两个子任务的相互益处。在框架内，稳健结构的2.5D姿势估计旨在基于深度关系识别人际遮挡。此外，我们开发了一种端到端几何感知的深度推理方法，该方法利用了2.5D姿势和以摄像头为中心的根深度的相互益处。该方法首先使用2.5D姿势和几何信息来推断向前通行证中以相机为中心的根深度，然后利用根深蒂固，以进一步改善向后通过的2.5D姿势估计的表示。此外，我们设计了一种自适应融合方案，该方案利用视觉感知和身体几何形状来减轻固有的深度歧义问题。广泛的实验证明了我们提出的模型比广泛的自下而上方法的优越性。我们的准确性甚至与自上而下的同行竞争。值得注意的是，我们的模型比现有的自下而上和自上而下的方法快得多。

尽管最近的进步，但是，尽管最近的进展，但是从单个图像中的人类姿势的全3D估计仍然是一个具有挑战性的任务。在本文中，我们探讨了关于场景几何体的强先前信息的假设可用于提高姿态估计精度。为了主弱地解决这个问题，我们已经组装了一种新的$ \ textbf {几何姿势提供} $ DataSet，包括与各种丰富的3D环境交互的人员的多视图图像。我们利用商业运动捕获系统来收集场景本身的姿势和构造精确的几何3D CAD模型的金标估计。要将对现有框架的现有框架注入图像的现有框架，我们介绍了一种新颖的，基于视图的场景几何形状，一个$ \ textbf {多层深度图} $，它采用了多次射线跟踪到简明地编码沿着每种相机视图光线方向的多个表面入口和退出点。我们提出了两种不同的机制，用于集成多层深度信息姿势估计：输入作为升降2D姿势的编码光线特征，其次是促进学习模型以支持几何一致姿态估计的可差异损失。我们通过实验展示这些技术可以提高3D姿势估计的准确性，特别是在遮挡和复杂场景几何形状的存在中。

在本文中，我们考虑了同时找到和从单个2D图像中恢复多手的具有挑战性的任务。先前的研究要么关注单手重建，要么以多阶段的方式解决此问题。此外，常规的两阶段管道首先检测到手部区域，然后估计每个裁剪贴片的3D手姿势。为了减少预处理和特征提取中的计算冗余，我们提出了一条简洁但有效的单阶段管道。具体而言，我们为多手重建设计了多头自动编码器结构，每个HEAD网络分别共享相同的功能图并分别输出手动中心，姿势和纹理。此外，我们采用了一个弱监督的计划来减轻昂贵的3D现实世界数据注释的负担。为此，我们提出了一系列通过舞台训练方案优化的损失，其中根据公开可用的单手数据集生成具有2D注释的多手数据集。为了进一步提高弱监督模型的准确性，我们在单手和多个手设置中采用了几个功能一致性约束。具体而言，从本地功能估算的每只手的关键点应与全局功能预测的重新投影点一致。在包括Freihand，HO3D，Interhand 2.6M和RHD在内的公共基准测试的广泛实验表明，我们的方法在弱监督和完全监督的举止中优于基于最先进的模型方法。代码和模型可在{\ url {https://github.com/zijinxuxu/smhr}}上获得。

This paper addresses the challenge of 6DoF pose estimation from a single RGB image under severe occlusion or truncation. Many recent works have shown that a two-stage approach, which first detects keypoints and then solves a Perspective-n-Point (PnP) problem for pose estimation, achieves remarkable performance. However, most of these methods only localize a set of sparse keypoints by regressing their image coordinates or heatmaps, which are sensitive to occlusion and truncation. Instead, we introduce a Pixel-wise Voting Network (PVNet) to regress pixel-wise unit vectors pointing to the keypoints and use these vectors to vote for keypoint locations using RANSAC. This creates a flexible representation for localizing occluded or truncated keypoints. Another important feature of this representation is that it provides uncertainties of keypoint locations that can be further leveraged by the PnP solver. Experiments show that the proposed approach outperforms the state of the art on the LINEMOD, Occlusion LINEMOD and YCB-Video datasets by a large margin, while being efficient for real-time pose estimation. We further create a Truncation LINEMOD dataset to validate the robustness of our approach against truncation. The code will be avaliable at https://zju-3dv.github.io/pvnet/.

We propose a single-shot approach for simultaneously detecting an object in an RGB image and predicting its 6D pose without requiring multiple stages or having to examine multiple hypotheses. Unlike a recently proposed single-shot technique for this task [11] that only predicts an approximate 6D pose that must then be refined, ours is accurate enough not to require additional post-processing. As a result, it is much faster -50 fps on a Titan X (Pascal) GPU -and more suitable for real-time processing. The key component of our method is a new CNN architecture inspired by [28,29] that directly predicts the 2D image locations of the projected vertices of the object's 3D bounding box. The object's 6D pose is then estimated using a PnP algorithm.For single object and multiple object pose estimation on the LINEMOD and OCCLUSION datasets, our approach substantially outperforms other recent 26] when they are all used without postprocessing. During post-processing, a pose refinement step can be used to boost the accuracy of these two methods, but at 10 fps or less, they are much slower than our method.

Realtime multi-person 2D pose estimation is a key component in enabling machines to have an understanding of people in images and videos. In this work, we present a realtime approach to detect the 2D pose of multiple people in an image. The proposed method uses a nonparametric representation, which we refer to as Part Affinity Fields (PAFs), to learn to associate body parts with individuals in the image. This bottom-up system achieves high accuracy and realtime performance, regardless of the number of people in the image. In previous work, PAFs and body part location estimation were refined simultaneously across training stages. We demonstrate that a PAF-only refinement rather than both PAF and body part location refinement results in a substantial increase in both runtime performance and accuracy. We also present the first combined body and foot keypoint detector, based on an internal annotated foot dataset that we have publicly released. We show that the combined detector not only reduces the inference time compared to running them sequentially, but also maintains the accuracy of each component individually. This work has culminated in the release of OpenPose, the first open-source realtime system for multi-person 2D pose detection, including body, foot, hand, and facial keypoints.

尽管单眼3D姿势估计似乎在公共数据集上取得了非常准确的结果，但它们的概括能力在很大程度上被忽略了。在这项工作中，我们对现有方法进行系统评估，并发现在对不同的摄像机，人体姿势和外观进行测试时，它们会出现更大的错误。为了解决这个问题，我们介绍了VirtualPose，这是一个两阶段的学习框架，以利用该任务特定的隐藏的“免费午餐”，即免费生成无限数量的姿势和摄像头，以免费培训模型。为此，第一阶段将图像转换为抽象的几何表示（AGR），然后第二阶段将它们映射到3D姿势。它从两个方面解决了概括问题：（1）可以在不同的2D数据集上对第一阶段进行培训，以降低过度合适外观的风险； （2）第二阶段可以接受从大量虚拟摄像机和姿势合成的不同AGR训练。它的表现优于SOTA方法，而无需使用任何配对的图像和3D姿势，从而为实用应用铺平了道路。代码可从https://github.com/wkom/virtualpose获得。

瑜伽是全球广受好评的，广泛推荐的健康生活实践。在执行瑜伽时保持正确的姿势至关重要。在这项工作中，我们采用了从人类姿势估计模型中的转移学习来提取整个人体的136个关键点，以训练一个随机的森林分类器，该分类器用于估算瑜伽室。在内部收集的内部收集的瑜伽视频数据库中评估了结果，该数据库是从4个不同的相机角度记录的51个主题。我们提出了一个三步方案，用于通过对1）看不见的帧，2）看不见的受试者进行测试来评估瑜伽分类器的普遍性。我们认为，对于大多数应用程序，对看不见的主题的验证精度和看不见的摄像头是最重要的。我们经验分析了三个公共数据集，转移学习的优势以及目标泄漏的可能性。我们进一步证明，分类精度在很大程度上取决于所采用的交叉验证方法，并且通常会产生误导。为了促进进一步的研究，我们已公开提供关键点数据集和代码。

实时机器人掌握，支持随后的精确反对操作任务，是高级高级自治系统的优先目标。然而，尚未找到这样一种可以用时间效率进行充分准确的掌握的算法。本文提出了一种新的方法，其具有2阶段方法，它使用深神经网络结合快速的2D对象识别，以及基于点对特征框架的随后的精确和快速的6D姿态估计来形成实时3D对象识别和抓握解决方案能够多对象类场景。所提出的解决方案有可能在实时应用上稳健地进行，需要效率和准确性。为了验证我们的方法，我们进行了广泛且彻底的实验，涉及我们自己的数据集的费力准备。实验结果表明，该方法在5CM5DEG度量标准中的精度97.37％，平均距离度量分数99.37％。实验结果显示了通过使用该方法的总体62％的相对改善（5cm5deg度量）和52.48％（平均距离度量）。此外，姿势估计执行也显示出运行时间的平均改善47.6％。最后，为了说明系统在实时操作中的整体效率，进行了一个拾取和放置的机器人实验，并显示了90％的准确度的令人信服的成功率。此实验视频可在https://sites.google.com/view/dl-ppf6dpose/上获得。

Estimating the 6D pose of known objects is important for robots to interact with the real world. The problem is challenging due to the variety of objects as well as the complexity of a scene caused by clutter and occlusions between objects. In this work, we introduce PoseCNN, a new Convolutional Neural Network for 6D object pose estimation. PoseCNN estimates the 3D translation of an object by localizing its center in the image and predicting its distance from the camera. The 3D rotation of the object is estimated by regressing to a quaternion representation. We also introduce a novel loss function that enables PoseCNN to handle symmetric objects. In addition, we contribute a large scale video dataset for 6D object pose estimation named the YCB-Video dataset. Our dataset provides accurate 6D poses of 21 objects from the YCB dataset observed in 92 videos with 133,827 frames. We conduct extensive experiments on our YCB-Video dataset and the OccludedLINEMOD dataset to show that PoseCNN is highly robust to occlusions, can handle symmetric objects, and provide accurate pose estimation using only color images as input. When using depth data to further refine the poses, our approach achieves state-of-the-art results on the challenging OccludedLINEMOD dataset. Our code and dataset are available at https://rse-lab.cs.washington.edu/projects/posecnn/.

We introduce MegaPose, a method to estimate the 6D pose of novel objects, that is, objects unseen during training. At inference time, the method only assumes knowledge of (i) a region of interest displaying the object in the image and (ii) a CAD model of the observed object. The contributions of this work are threefold. First, we present a 6D pose refiner based on a render&compare strategy which can be applied to novel objects. The shape and coordinate system of the novel object are provided as inputs to the network by rendering multiple synthetic views of the object's CAD model. Second, we introduce a novel approach for coarse pose estimation which leverages a network trained to classify whether the pose error between a synthetic rendering and an observed image of the same object can be corrected by the refiner. Third, we introduce a large-scale synthetic dataset of photorealistic images of thousands of objects with diverse visual and shape properties and show that this diversity is crucial to obtain good generalization performance on novel objects. We train our approach on this large synthetic dataset and apply it without retraining to hundreds of novel objects in real images from several pose estimation benchmarks. Our approach achieves state-of-the-art performance on the ModelNet and YCB-Video datasets. An extensive evaluation on the 7 core datasets of the BOP challenge demonstrates that our approach achieves performance competitive with existing approaches that require access to the target objects during training. Code, dataset and trained models are available on the project page: https://megapose6d.github.io/.

人类的姿势估计旨在弄清不同场景中所有人的关键。尽管结果有希望，但目前的方法仍然面临一些挑战。现有的自上而下的方法单独处理一个人，而没有不同的人与所在的场景之间的相互作用。因此，当发生严重闭塞时，人类检测的表现会降低。另一方面，现有的自下而上方法同时考虑所有人，并捕获整个图像的全局知识。但是，由于尺度变化，它们的准确性不如自上而下的方法。为了解决这些问题，我们通过整合自上而下和自下而上的管道来探索不同接受场的视觉线索并实现其互补性，提出了一种新颖的双皮线整合变压器（DPIT）。具体而言，DPIT由两个分支组成，自下而上的分支介绍了整个图像以捕获全局视觉信息，而自上而下的分支则从单人类边界框中提取本地视觉的特征表示。然后，从自下而上和自上而下的分支中提取的特征表示形式被馈入变压器编码器，以交互融合全局和本地知识。此外，我们定义了关键点查询，以探索全景和单人类姿势视觉线索，以实现两个管道的相互互补性。据我们所知，这是将自下而上和自上而下管道与变压器与人类姿势估计的变压器相结合的最早作品之一。关于可可和MPII数据集的广泛实验表明，我们的DPIT与最先进的方法相当。

The detection of human body and its related parts (e.g., face, head or hands) have been intensively studied and greatly improved since the breakthrough of deep CNNs. However, most of these detectors are trained independently, making it a challenging task to associate detected body parts with people. This paper focuses on the problem of joint detection of human body and its corresponding parts. Specifically, we propose a novel extended object representation that integrates the center location offsets of body or its parts, and construct a dense single-stage anchor-based Body-Part Joint Detector (BPJDet). Body-part associations in BPJDet are embedded into the unified representation which contains both the semantic and geometric information. Therefore, BPJDet does not suffer from error-prone association post-matching, and has a better accuracy-speed trade-off. Furthermore, BPJDet can be seamlessly generalized to jointly detect any body part. To verify the effectiveness and superiority of our method, we conduct extensive experiments on the CityPersons, CrowdHuman and BodyHands datasets. The proposed BPJDet detector achieves state-of-the-art association performance on these three benchmarks while maintains high accuracy of detection. Code will be released to facilitate further studies.

The goal of this paper is to estimate the 6D pose and dimensions of unseen object instances in an RGB-D image. Contrary to "instance-level" 6D pose estimation tasks, our problem assumes that no exact object CAD models are available during either training or testing time. To handle different and unseen object instances in a given category, we introduce Normalized Object Coordinate Space (NOCS)-a shared canonical representation for all possible object instances within a category. Our region-based neural network is then trained to directly infer the correspondence from observed pixels to this shared object representation (NOCS) along with other object information such as class label and instance mask. These predictions can be combined with the depth map to jointly estimate the metric 6D pose and dimensions of multiple objects in a cluttered scene. To train our network, we present a new contextaware technique to generate large amounts of fully annotated mixed reality data. To further improve our model and evaluate its performance on real data, we also provide a fully annotated real-world dataset with large environment and instance variation. Extensive experiments demonstrate that the proposed method is able to robustly estimate the pose and size of unseen object instances in real environments while also achieving state-of-the-art performance on standard 6D pose estimation benchmarks.