观察噪声，由于对称性和遮挡导致的分割和模糊性导致不准确的对象姿态估计。虽然深度和基于RGB的姿势细化方法提高了所得姿势估计的准确性，但它们在观察中易受歧义的影响，因为它们考虑了视觉对齐。我们建议利用我们经常观察静态，刚性场景的事实。因此，其中的对象需要在物理上似是姿势。我们表明，考虑合理性降低了歧义，结果，允许在杂乱的环境中更准确地预测姿势。为此，我们扩展了最近基于RL的登记方法，以实现对象姿势的迭代细化。LineMod和YCB-Video数据集的实验证明了我们深度的细化方法的最先进的性能。

Estimating 6D poses of objects from images is an important problem in various applications such as robot manipulation and virtual reality. While direct regression of images to object poses has limited accuracy, matching rendered images of an object against the input image can produce accurate results. In this work, we propose a novel deep neural network for 6D pose matching named DeepIM. Given an initial pose estimation, our network is able to iteratively refine the pose by matching the rendered image against the observed image. The network is trained to predict a relative pose transformation using a disentangled representation of 3D location and 3D orientation and an iterative training process. Experiments on two commonly used benchmarks for 6D pose estimation demonstrate that DeepIM achieves large improvements over stateof-the-art methods. We furthermore show that DeepIM is able to match previously unseen objects.

We introduce an approach for recovering the 6D pose of multiple known objects in a scene captured by a set of input images with unknown camera viewpoints. First, we present a single-view single-object 6D pose estimation method, which we use to generate 6D object pose hypotheses. Second, we develop a robust method for matching individual 6D object pose hypotheses across different input images in order to jointly estimate camera viewpoints and 6D poses of all objects in a single consistent scene. Our approach explicitly handles object symmetries, does not require depth measurements, is robust to missing or incorrect object hypotheses, and automatically recovers the number of objects in the scene. Third, we develop a method for global scene refinement given multiple object hypotheses and their correspondences across views. This is achieved by solving an object-level bundle adjustment problem that refines the poses of cameras and objects to minimize the reprojection error in all views. We demonstrate that the proposed method, dubbed Cosy-Pose, outperforms current state-of-the-art results for single-view and multi-view 6D object pose estimation by a large margin on two challenging benchmarks: the YCB-Video and T-LESS datasets. Code and pre-trained models are available on the project webpage. 5

A key technical challenge in performing 6D object pose estimation from RGB-D image is to fully leverage the two complementary data sources. Prior works either extract information from the RGB image and depth separately or use costly post-processing steps, limiting their performances in highly cluttered scenes and real-time applications. In this work, we present DenseFusion, a generic framework for estimating 6D pose of a set of known objects from RGB-D images. DenseFusion is a heterogeneous architecture that processes the two data sources individually and uses a novel dense fusion network to extract pixel-wise dense feature embedding, from which the pose is estimated. Furthermore, we integrate an end-to-end iterative pose refinement procedure that further improves the pose estimation while achieving near real-time inference. Our experiments show that our method outperforms state-of-the-art approaches in two datasets, YCB-Video and LineMOD. We also deploy our proposed method to a real robot to grasp and manipulate objects based on the estimated pose. Our code and video are available at https://sites.google.com/view/densefusion/.

我们提出了一种称为DPODV2（密集姿势对象检测器）的三个阶段6 DOF对象检测方法，该方法依赖于致密的对应关系。我们将2D对象检测器与密集的对应关系网络和多视图姿势细化方法相结合，以估计完整的6 DOF姿势。与通常仅限于单眼RGB图像的其他深度学习方法不同，我们提出了一个统一的深度学习网络，允许使用不同的成像方式（RGB或DEPTH）。此外，我们提出了一种基于可区分渲染的新型姿势改进方法。主要概念是在多个视图中比较预测并渲染对应关系，以获得与所有视图中预测的对应关系一致的姿势。我们提出的方法对受控设置中的不同数据方式和培训数据类型进行了严格的评估。主要结论是，RGB在对应性估计中表现出色，而如果有良好的3D-3D对应关系，则深度有助于姿势精度。自然，他们的组合可以实现总体最佳性能。我们进行广泛的评估和消融研究，以分析和验证几个具有挑战性的数据集的结果。 DPODV2在所有这些方面都取得了出色的成果，同时仍然保持快速和可扩展性，独立于使用的数据模式和培训数据的类型

在这项工作中，我们解决了共同跟踪手对象姿势并从野外深度点云序列重建形状的具有挑战性，HandTrackNet，以估计框架间的手动运动。我们的HandTrackNet提出了一个新型的手姿势构成典型化模块，以简化跟踪任务，从而产生准确且稳健的手工关节跟踪。然后，我们的管道通过将预测的手关节转换为基于模板的参数手模型mano来重建全手。对于对象跟踪，我们设计了一个简单而有效的模块，该模块从第一帧估算对象SDF并执行基于优化的跟踪。最后，采用联合优化步骤执行联合手和物体推理，从而减轻了闭塞引起的歧义并进一步完善了手姿势。在训练过程中，整个管道仅看到纯粹的合成数据，这些数据与足够的变化并通过深度模拟合成，以易于概括。整个管道与概括差距有关，因此可以直接传输到真实的野外数据。我们在两个真实的手对象交互数据集上评估我们的方法，例如HO3D和DEXYCB，没有任何填充。我们的实验表明，所提出的方法显着优于先前基于深度的手和对象姿势估计和跟踪方法，以9 fps的帧速率运行。

虽然最近出现了类别级的9DOF对象姿势估计，但由于较大的对象形状和颜色等类别内差异，因此，先前基于对应的或直接回归方法的准确性均受到限制。 - 级别的物体姿势和尺寸炼油机Catre，能够迭代地增强点云的姿势估计以产生准确的结果。鉴于初始姿势估计，Catre通过对齐部分观察到的点云和先验的抽象形状来预测初始姿势和地面真理之间的相对转换。具体而言，我们提出了一种新颖的分离体系结构，以了解旋转与翻译/大小估计之间的固有区别。广泛的实验表明，我们的方法在REAL275，Camera25和LM基准测试中的最先进方法高达〜85.32Hz，并在类别级别跟踪上取得了竞争成果。我们进一步证明，Catre可以对看不见的类别进行姿势改进。可以使用代码和训练有素的型号。

透明对象对视觉感知系统提出了多个不同的挑战。首先，他们缺乏区分视觉特征使透明对象比不透明的对象更难检测和本地化。即使人类也发现某些透明的表面几乎没有镜面反射或折射，例如玻璃门，难以感知。第二个挑战是，通常用于不透明对象感知的常见深度传感器由于其独特的反射特性而无法对透明对象进行准确的深度测量。由于这些挑战，我们观察到，同一类别（例如杯子）内的透明对象实例看起来与彼此相似，而不是同一类别的普通不透明对象。鉴于此观察结果，本文着手探讨类别级透明对象姿势估计的可能性，而不是实例级姿势估计。我们提出了TransNet，这是一种两阶段的管道，该管道学会使用局部深度完成和表面正常估计来估计类别级别的透明对象姿势。在最近的大规模透明对象数据集中，根据姿势估计精度评估了TransNet，并将其与最先进的类别级别姿势估计方法进行了比较。该比较的结果表明，TransNet可以提高透明对象的姿势估计准确性，并从随附的消融研究中提高了关键发现，这表明未来的方向改善了绩效。

We introduce MegaPose, a method to estimate the 6D pose of novel objects, that is, objects unseen during training. At inference time, the method only assumes knowledge of (i) a region of interest displaying the object in the image and (ii) a CAD model of the observed object. The contributions of this work are threefold. First, we present a 6D pose refiner based on a render&compare strategy which can be applied to novel objects. The shape and coordinate system of the novel object are provided as inputs to the network by rendering multiple synthetic views of the object's CAD model. Second, we introduce a novel approach for coarse pose estimation which leverages a network trained to classify whether the pose error between a synthetic rendering and an observed image of the same object can be corrected by the refiner. Third, we introduce a large-scale synthetic dataset of photorealistic images of thousands of objects with diverse visual and shape properties and show that this diversity is crucial to obtain good generalization performance on novel objects. We train our approach on this large synthetic dataset and apply it without retraining to hundreds of novel objects in real images from several pose estimation benchmarks. Our approach achieves state-of-the-art performance on the ModelNet and YCB-Video datasets. An extensive evaluation on the 7 core datasets of the BOP challenge demonstrates that our approach achieves performance competitive with existing approaches that require access to the target objects during training. Code, dataset and trained models are available on the project page: https://megapose6d.github.io/.

We present a novel method for detecting 3D model instances and estimating their 6D poses from RGB data in a single shot. To this end, we extend the popular SSD paradigm to cover the full 6D pose space and train on synthetic model data only. Our approach competes or surpasses current state-of-the-art methods that leverage RGB-D data on multiple challenging datasets. Furthermore, our method produces these results at around 10Hz, which is many times faster than the related methods. For the sake of reproducibility, we make our trained networks and detection code publicly available. 1

视觉感知任务通常需要大量的标记数据，包括3D姿势和图像空间分割掩码。创建此类培训数据集的过程可能很难或耗时，可以扩展到一般使用的功效。考虑对刚性对象的姿势估计的任务。在大型公共数据集中接受培训时，基于神经网络的深层方法表现出良好的性能。但是，将这些网络调整为其他新颖对象，或针对不同环境的现有模型进行微调，需要大量的时间投资才能产生新标记的实例。为此，我们提出了ProgressLabeller作为一种方法，以更有效地以可扩展的方式从彩色图像序列中生成大量的6D姿势训练数据。 ProgressLabeller还旨在支持透明或半透明的对象，以深度密集重建的先前方法将失败。我们通过快速创建一个超过1M样品的数据集来证明ProgressLabeller的有效性，我们将其微调一个最先进的姿势估计网络，以显着提高下游机器人的抓地力。 ProgressLabeller是https://github.com/huijiezh/progresslabeller的开放源代码。

本文提出了一种类别级别的6D对象姿势和形状估计方法IDAPS，其允许在类别中跟踪6D姿势并估计其3D形状。我们使用深度图像作为输入开发类别级别自动编码器网络，其中来自自动编码器编码的特征嵌入在类别中对象的姿势。自动编码器可用于粒子过滤器框架，以估计和跟踪类别中的对象的姿势。通过利用基于符号距离函数的隐式形状表示，我们构建延迟网络以估计给定对象的估计姿势的3D形状的潜在表示。然后，估计的姿势和形状可用于以迭代方式互相更新。我们的类别级别6D对象姿势和形状估计流水线仅需要2D检测和分段进行初始化。我们在公开的数据集中评估我们的方法，并展示其有效性。特别是，我们的方法在形状估计上实现了相对高的准确性。

The goal of this paper is to estimate the 6D pose and dimensions of unseen object instances in an RGB-D image. Contrary to "instance-level" 6D pose estimation tasks, our problem assumes that no exact object CAD models are available during either training or testing time. To handle different and unseen object instances in a given category, we introduce Normalized Object Coordinate Space (NOCS)-a shared canonical representation for all possible object instances within a category. Our region-based neural network is then trained to directly infer the correspondence from observed pixels to this shared object representation (NOCS) along with other object information such as class label and instance mask. These predictions can be combined with the depth map to jointly estimate the metric 6D pose and dimensions of multiple objects in a cluttered scene. To train our network, we present a new contextaware technique to generate large amounts of fully annotated mixed reality data. To further improve our model and evaluate its performance on real data, we also provide a fully annotated real-world dataset with large environment and instance variation. Extensive experiments demonstrate that the proposed method is able to robustly estimate the pose and size of unseen object instances in real environments while also achieving state-of-the-art performance on standard 6D pose estimation benchmarks.

我们提出了一种方法，用于估计具有单个RGB图像的可用3D模型的刚性对象的6DOF姿势。与基于经典对应的方法不同，该方法可以预测输入图像的像素的3D对象坐标，该建议的方法可以预测3D对象坐标在相机frustum中采样的3D查询点。从像素到3D点的移动，这是受到3D重建方法的最新PIFU式方法的启发，可以对整个对象（包括（自我）遮挡部分）进行推理。对于与与像素对齐的图像功能相关的3D查询点，我们训练完全连接的神经网络来预测：（i）相应的3D对象坐标，以及（ii）签名到对象表面的签名距离，首先定义仅适用于地表附近的查询点。我们将该网络实现的映射称为神经通信字段。然后，通过Kabsch-Ransac算法从预测的3D-3D对应关系中稳健地估计对象姿势。所提出的方法在三个BOP数据集上实现了最先进的结果，并且在咬合挑战性案例中表现出了优越。项目网站在：linhuang17.github.io/ncf。

估计对象的6D姿势是必不可少的计算机视觉任务。但是，大多数常规方法从单个角度依赖相机数据，因此遭受遮挡。我们通过称为MV6D的新型多视图6D姿势估计方法克服了这个问题，该方法从多个角度根据RGB-D图像准确地预测了混乱场景中所有对象的6D姿势。我们将方法以PVN3D网络为基础，该网络使用单个RGB-D图像来预测目标对象的关键点。我们通过从多个视图中使用组合点云来扩展此方法，并将每个视图中的图像与密集层层融合。与当前的多视图检测网络（例如Cosypose）相反，我们的MV6D可以以端到端的方式学习多个观点的融合，并且不需要多个预测阶段或随后对预测的微调。此外，我们介绍了三个新颖的影像学数据集，这些数据集具有沉重的遮挡的混乱场景。所有这些都从多个角度包含RGB-D图像，例如语义分割和6D姿势估计。即使在摄像头不正确的情况下，MV6D也明显优于多视图6D姿势估计中最新的姿势估计。此外，我们表明我们的方法对动态相机设置具有强大的态度，并且其准确性随着越来越多的观点而逐渐增加。

We introduce a novel method for 3D object detection and pose estimation from color images only. We first use segmentation to detect the objects of interest in 2D even in presence of partial occlusions and cluttered background. By contrast with recent patch-based methods, we rely on a "holistic" approach: We apply to the detected objects a Convolutional Neural Network (CNN) trained to predict their 3D poses in the form of 2D projections of the corners of their 3D bounding boxes. This, however, is not sufficient for handling objects from the recent T-LESS dataset: These objects exhibit an axis of rotational symmetry, and the similarity of two images of such an object under two different poses makes training the CNN challenging. We solve this problem by restricting the range of poses used for training, and by introducing a classifier to identify the range of a pose at run-time before estimating it. We also use an optional additional step that refines the predicted poses. We improve the state-of-the-art on the LINEMOD dataset from 73.7% [2] to 89.3% of correctly registered RGB frames. We are also the first to report results on the Occlusion dataset [1] using color images only. We obtain 54% of frames passing the Pose 6D criterion on average on several sequences of the T-LESS dataset, compared to the 67% of the state-of-the-art [10] on the same sequences which uses both color and depth. The full approach is also scalable, as a single network can be trained for multiple objects simultaneously.

RGB图像的刚性对象的可伸缩6D构成估计旨在处理多个对象并推广到新物体。我们建立在一个著名的自动编码框架的基础上，以应对对象对称性和缺乏标记的训练数据，我们通过将自动编码器的潜在表示形状分解为形状并构成子空间来实现可伸缩性。潜在形状空间通过对比度度量学习模型不同对象的相似性，并将潜在姿势代码与旋转检索的规范旋转进行比较。由于不同的对象对称会诱导不一致的潜在姿势空间，因此我们用规范旋转重新输入形状表示，以生成形状依赖的姿势代码簿以进行旋转检索。我们在两个基准上显示了最新的性能，其中包含无类别和每日对象的无纹理CAD对象，并通过扩展到跨类别的每日对象的更具挑战性的设置，进一步证明了可扩展性。

Generating grasp poses is a crucial component for any robot object manipulation task. In this work, we formulate the problem of grasp generation as sampling a set of grasps using a variational autoencoder and assess and refine the sampled grasps using a grasp evaluator model. Both Grasp Sampler and Grasp Refinement networks take 3D point clouds observed by a depth camera as input. We evaluate our approach in simulation and real-world robot experiments. Our approach achieves 88% success rate on various commonly used objects with diverse appearances, scales, and weights. Our model is trained purely in simulation and works in the real world without any extra steps. The video of our experiments can be found here.

We propose a single-shot approach for simultaneously detecting an object in an RGB image and predicting its 6D pose without requiring multiple stages or having to examine multiple hypotheses. Unlike a recently proposed single-shot technique for this task [11] that only predicts an approximate 6D pose that must then be refined, ours is accurate enough not to require additional post-processing. As a result, it is much faster -50 fps on a Titan X (Pascal) GPU -and more suitable for real-time processing. The key component of our method is a new CNN architecture inspired by [28,29] that directly predicts the 2D image locations of the projected vertices of the object's 3D bounding box. The object's 6D pose is then estimated using a PnP algorithm.For single object and multiple object pose estimation on the LINEMOD and OCCLUSION datasets, our approach substantially outperforms other recent 26] when they are all used without postprocessing. During post-processing, a pose refinement step can be used to boost the accuracy of these two methods, but at 10 fps or less, they are much slower than our method.

微弱的物理是计算机视觉和机器人的强大工具，用于了解互动的场景理解和推理。现有方法经常被限于具有预先已知的简单形状或形状的物体。在本文中，我们提出了一种新的方法来具有摩擦触点的可分解物理学，其利用符号距离场（SDF）隐含地表示物理形状。我们的模拟即使涉及的形状为非凸形表示，也支持接触点计算。此外，我们提出了区分对象形状的动力学来利用基于梯度的方法来促进形状优化。在我们的实验中，我们证明我们的方法允许从轨迹和深度图像观察的诸如摩擦系数，质量，力或形状参数的物理参数的基于模型的推断，并且在几个具有挑战性的合成场景和真实图像序列中。